VERITAS: Towards a General-Purpose Replication Tool for Scientific Research
06:00 · July 7, 2026 · arXiv cs.AI RSS
AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.
Summary
VERITAS addresses the widening gap between rapid AI-driven scientific publication and the capacity of traditional peer review to verify results. The framework provides a general-purpose pipeline that accepts a paper, an associated code repository, or both, then uses CLI coding agents to extract empirical claims, implement or adapt the described methodology, resolve runtime issues, and produce a structured replication verdict.
The system operates in six phases. An analyze step converts the input into typed claims annotated with importance levels. When no repository is supplied, a dedicated codegen phase produces an initial implementation from the paper alone. Subsequent plan, replicate, assess-fixes, and verify stages draft an execution procedure, run experiments while logging every intervention, rate the severity of those interventions, and compare observed outcomes against the extracted claims. The final output is an importance-weighted Replication Score together with a severity-rated fix log and the patched codebase.
VERITAS supports three input modes—full (paper plus repository), paper-only, and repo-only—and keeps the claim set hidden from the replication agent to reduce leakage. It has been evaluated on CORE-Bench and ReplicationBench, covering 65 papers across computer science, medicine, social science, and astrophysics. On identical model and host environments, it surpasses two strong Claude Code baselines on every reported metric, including per-capsule pass rate, per-task match rate, and adapted measures of trajectory faithfulness and unauthorized source access.
Why it matters
Directly addresses reproducibility and verification needs for Dutch AI researchers and labs; offers actionable tooling that can be adopted in NL/EU academic and industrial settings focused on trustworthy AI.
