ASI-Bench: Measuring AI Agents' Scientific Autonomy Without Method Hints

Zhou, Sun et al.; corresponding author Yongchao Chen (Tsinghua + collaborators) · August 2026 · 60 tasks · 11 domains · 18 agent×model configs
arXiv 2608.17271Podcast

References

  1. ASI-Bench: At the Dawn of Artificial Superintelligence — Zhou, Sun et al., 2026 · arXiv 2608.17271
  2. PaperBench — Starace, Jaffe et al., 2025 · arXiv 2504.01848
  3. MLE-bench — Chan, Chowdhury et al., 2024 · arXiv 2410.07095
  4. SWE-bench — Jimenez, Yang et al., 2023 · arXiv 2310.06770
  5. ScienceAgentBench — Chen et al., 2024 · arXiv 2410.05080
  6. RE-Bench — Wijk et al., 2024 · arXiv 2411.15114
  7. Chain-of-Thought Prompting — Wei et al., 2022 · arXiv 2201.11903
  8. LLMs Can Be Easily Distracted by Irrelevant Context — Shi et al., 2023 · arXiv 2302.00093
  9. Also discussed: AI Scientist v1/v2, ProjectionBench, ProjectEval, FIRE-Bench / ResearchClawBench, SWE-bench Pro (see Scholar)