FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
06:00 · July 8, 2026 · arXiv cs.AI RSS
LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.
Summary
FirstResearch addresses a persistent gap in LLM-driven scientific discovery: while current agents can generate plausible research questions, those outputs often conceal the underlying assumptions, mechanisms, and falsifying conditions that human scientists need to inspect. The framework enforces a structured derivation process that begins with explicit primitive definitions and first-principles assumptions, then builds a mechanism model, surfaces tensions or contradictions, and produces a Research Question Certificate for each candidate. This certificate records a falsifiable hypothesis, a minimal decisive test, expected observations, and a failure-update rule, allowing the question to be audited before any downstream experiment design or execution begins.
A novelty-aware gate further refines weak but formally valid certificates by requiring mechanism-boundary signals such as thresholds, interactions, or failure regimes. On a benchmark of ten LLM-agent research topics, the system was compared against controlled prompt-level baselines modeled on AI co-scientist, Agent Laboratory, and AI Scientist-v2. Under a primary DeepSeek judge, FirstResearch produced higher average rubric scores; an independent Gemini-2.5-Flash rescore preserved the ranking, with FirstResearch at 4.86/5 versus 4.38/5 for the strongest baseline and strong inter-judge agreement. An ablation isolating the certificate component confirmed its central contribution, while removal of the certificate caused scores to collapse.
The authors emphasize that the results remain preliminary because they rely on LLM judges rather than domain-expert human review, and they position FirstResearch as a narrow, complementary layer rather than a full autonomous research pipeline. All code, prompts, and reproduction scripts are released publicly to support further validation of explicit derivation constraints as a route to more auditable LLM-generated scientific questions.
Why it matters
This research is highly relevant to the Dutch AI market's focus on transparent and ethical AI. By making LLM-generated scientific hypotheses auditable and inspectable, it aligns with EU regulatory priorities and offers Dutch researchers a robust tool for accountable AI-driven scientific discovery.
