ASI-Bench: At the Dawn of Artificial Superintelligence
06:00 · August 19, 2026 · arXiv cs.AI RSS
Artificial superintelligence (ASI) requires AI to move beyond mastering existing knowledge toward exploring the unknown, creating new knowledge, and turning new ideas into verifiable results. However, the capabilities of today's AI systems are still largely built on learning, compressing, and applying existing human knowledge. Accordingly, existing benchmarks primarily test whether AI can produce correct answers based on learned knowledge, or whether it can complete tasks under extensive human guidance. We therefore introduce ASI-Bench, the first benchmark to jointly evaluate AI systems' capabilities of innovative exploration and autonomous scientific execution across general research domains, and the first to progressively withdraw human methodological guidance within the same research project to test how far AI can proceed on its own. Built by over 40 experts with the cost of 31,000+ human hours, ASI-Bench contains 60 project-level research tasks across 11 scientific domains and progressively reduces methodological guidance to test whether AI can independently select methods, conduct research, and produce verifiable results. All tasks undergo expert review, AI-assisted auditing, sandbox execution, and scorer validation. Across 18 state-of-the-art agent--model configurations, the average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified and 26.62 when agents must determine the method themselves. This sharp decline shows that current systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research. ASI-Bench is open to the world. We invite researchers and builders everywhere to contribute new tasks, challenge the limits of today's AI, and help accelerate humanity's collective path toward artificial superintelligence at https://asibench.apexin.ai/submit.
Summary
ASI-Bench addresses a core limitation in current AI evaluation: most existing benchmarks measure performance on tasks with known answers or under heavy human-specified procedures, leaving open the question of whether systems can independently explore unfamiliar scientific problems and produce verifiable results. The new benchmark introduces 60 project-level tasks spanning 11 domains, each defined by a research objective, domain-specific data, an executable environment, and reference artifacts that can be checked automatically. Tasks were constructed by more than 40 domain experts over 31,000 human hours and subjected to expert review, AI-assisted auditing, sandbox execution, and scorer validation to ensure scientific soundness and reliable scoring.
To isolate autonomy, ASI-Bench applies a controlled reduction of methodological guidance within the same projects. In the most guided setting, agents receive complete method descriptions; in subsequent settings only the method name is supplied, then only the research goal and data, and finally the same conditions with added distractors. Across 18 agent–model combinations, average scores fall from 50.91 under full guidance to 29.10 when only the method is named and 26.62 when the method itself must be chosen, indicating that current systems remain strongly dependent on explicit human direction for end-to-end scientific execution.
The benchmark is released with full reproducibility protocols, including sandbox environments and scoring code, and is structured to accept community-submitted tasks. This design supplies a shared reference for tracking progress from systems that primarily compress and apply existing knowledge toward those capable of sustained, autonomous scientific discovery.
Why it matters
Offers a novel, high-depth evaluation framework that Dutch AI researchers and advanced labs can directly apply to measure progress toward autonomous scientific agents, aligning with the Netherlands' strengths in ethical AI and SME-driven innovation.
