The Agentic Garden of Forking Paths
06:00 · July 3, 2026 · arXiv cs.AI RSS
Empirical research rarely admits a unique analysis. Different analytical choices can lead to different conclusions from the same data, yet these hidden forking paths are difficult to observe. We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit. Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions from the same data and question, with findings systematically aligned with those beliefs. In a study in which 42 human research teams analyzed the same immigration dataset, AI agents reproduced 72% of the human ideological gap in reported effect estimates. Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports: 86% passed independent AI review and 78% passed majority human expert review. These findings suggest that the central challenge is often not flawed analyses, but selective exploration and reporting from a large space of methodologically defensible analyses. AI agents may amplify this longstanding problem by making such exploration inexpensive and scalable. To address this, we introduce the m-value (multiverse value), the probability that an analysis path would produce a claim at least as extreme as the reported one. We further introduce Agentic Bootstrap, which estimates the m-value by using AI agents to sample plausible analysis paths. Applied to the human immigration study, 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m<0.05). Scientific evidence should therefore be evaluated not only by a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. Agentic Bootstrap makes this distribution observable and turns it into a criterion for scientific credibility.
Summary
Empirical research seldom yields a single defensible analysis. Different choices in data preparation, model specification, and variable selection routinely produce divergent conclusions from identical inputs, a pattern long described as the garden of forking paths. The paper demonstrates that large-language-model agents, when assigned distinct personas, reproduce much of this human variation while rendering the paths explicit and inspectable.
In controlled experiments spanning four high-stakes domains, persona-conditioned agents reached opposing conclusions on the same data and question, with results systematically aligned to the assigned ideological or disciplinary priors. When the same immigration dataset previously examined by 42 human research teams was given to agents, they recovered 72 percent of the ideological gap observed in the human effect estimates. Strikingly, the resulting agent reports proved difficult to dismiss on methodological grounds: 86 percent passed independent AI review and 78 percent passed majority review by human experts.
These findings indicate that the central difficulty is rarely outright analytical error but rather selective exploration within a large space of methodologically acceptable paths. Because agents can traverse that space at low cost, the authors argue they risk amplifying rather than mitigating selective reporting. To counter this, the paper introduces the m-value, defined as the probability that a randomly sampled defensible analysis would yield a result at least as extreme as the one reported. An accompanying procedure, the Agentic Bootstrap, estimates this quantity by directing agents to sample plausible analysis paths and thereby locate any single finding within the broader distribution of credible alternatives. In the immigration study, 13.5 percent of the human-reported analyses fell in the most extreme 5 percent of that distribution (m < 0.05). The authors conclude that scientific claims should be assessed not only by the quality of the reported path but also by their position within the observable space of alternatives.
Why it matters
This research is highly relevant for Dutch AI researchers and data scientists focused on transparent and robust AI methodologies. The introduction of the 'm-value' and 'Agentic Bootstrap' provides actionable tools to mitigate bias and improve the credibility of empirical research, aligning with the EU's strong emphasis on ethical AI practices.
