Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
06:00 · August 3, 2026 · arXiv cs.AI RSS
AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery. However, evaluating and comparing the quality of AI-generated papers remains an open challenge. We propose and implement a rigorous benchmarking protocol using an automated peer-review system that harnesses frontier large language models to assess scientific papers across four core dimensions: originality, scientific rigor, clarity, and significance. We evaluate four leading AI Scientist frameworks: \textit{Sakana AI (v1 & v2)}, \textit{CycleResearcher}, and \textit{Data-to-Paper}. Each framework was run on a consistent set of 15 research proposals published by a commercial autonomous AI scientist company (FARS), generating 60 papers that we evaluate alongside 15 FARS benchmark papers. Using three independent LLM reviewers (GPT-5.4, Gemini, and Claude), we find that FARS benchmark papers significantly outperform all competing frameworks, achieving mean scores of 2.14--2.47 on a 1--5 scale compared to 1.00--1.87 for other systems. Notably, FARS scores are more than 2$\times$ higher than the next-best systems on Gemini and Claude evaluations. We find strong agreement among Gemini and Claude ($\rho$ = 0.907, $p < 0.001$), and both correlate extremely strongly with the synthesis score ($\rho$ = 0.961, $p < 0.001$), validating the reliability of automated evaluation. However, GPT-5.4 exhibits weaker agreement ($\rho \approx 0.32$), suggesting it evaluates papers using different criteria. These results establish the first quantitative benchmark for AI Scientist systems and demonstrate that multi-model LLM evaluation provides a scalable, consistent framework for assessing autonomous research quality.
Summary
AI Scientist systems aim to automate the full research pipeline from hypothesis generation through experimentation and manuscript writing, yet comparing their output quality has lacked controlled, quantitative methods. This study addresses that gap by running four frameworks—Sakana AI v1 and v2, CycleResearcher, and Data-to-Paper—on the same set of 15 research proposals drawn from the FARS dataset, producing 60 papers that were then evaluated alongside 15 reference papers generated by FARS itself.
Evaluation relied on an automated multi-model protocol in which three frontier LLMs (GPT-5.4, Gemini, and Claude) independently scored every paper on four dimensions—originality, scientific rigor, clarity, and significance—using a 1–5 scale. The resulting dataset of 75 papers therefore permits direct, apples-to-apples comparison across systems that differ in architecture, from linear pipelines to agentic tree search and multi-agent shared workspaces.
Results show FARS papers achieving mean scores of 2.14–2.47, more than twice as high as the next-best systems on Gemini and Claude ratings. Gemini and Claude exhibited strong inter-rater agreement (ρ = 0.907) and aligned closely with an overall synthesis score (ρ = 0.961), supporting the reliability of the automated review process. GPT-5.4 displayed markedly weaker correlation (ρ ≈ 0.32), indicating divergent evaluation criteria.
By establishing the first controlled quantitative benchmark for autonomous research generation and demonstrating that multi-LLM review can scale while maintaining consistency, the work supplies both a practical evaluation framework and concrete performance baselines against which future AI Scientist systems can be measured.
Why it matters
Provides actionable benchmarking methodology and performance baselines that Dutch AI research groups can adopt or replicate when developing or selecting autonomous research tools; addresses reproducibility and evaluation challenges central to trustworthy AI deployment in the EU context.
