Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking
06:00 · August 19, 2026 · arXiv cs.AI RSS
Large language models (LLMs) are increasingly used for scientific hypothesis generation. However, evaluating generated hypotheses remains a challenge for trustworthy AI-enabled scientific workflows. Existing approaches often use LLMs as judges or rely on semantic similarity, which can favor familiar ideas over novel ones. We propose a logit-based energy scoring method that evaluates hypotheses using a language model's intrinsic confidence rather than comparative judgment. We benchmarked seven language models on 1,323 papers across 12 disciplines. Each paper was paired with its hypothesis and fifteen incorrect alternatives. Intrinsic scoring reached 33.0% Hit@1 pooled across both scorers, compared with 16.6% for prompted listwise ranking. The strongest configuration, a 1-billion-parameter model using logit-based energy scoring, reached 53.1%, though this was the maximum across 14 model-by-scorer combinations selected post hoc. Overall, intrinsic model confidence shows potential for scientific hypothesis evaluation. This study also motivates future research on confidence-based methods for trustworthy AI-enabled scientific discovery.
Summary
Large language models are now routinely enlisted to generate candidate hypotheses from scientific literature, yet reliable ways to rank those candidates remain underdeveloped. Prompt-based LLM-as-judge approaches and semantic-similarity metrics often favor conventional or previously seen formulations, introducing position and verbosity biases that can sideline genuinely novel ideas. The present work therefore examines an alternative that dispenses with explicit verbal judgment altogether.
Instead of prompting a model to compare hypotheses, the authors score each candidate directly from the model’s output-layer logits. They define an energy function based on the raw token probabilities assigned to the hypothesis text given the paper’s background and research question; lower energy indicates higher intrinsic compatibility. This approach is evaluated on a dataset of 1,323 papers spanning twelve disciplines. For each paper the correct hypothesis is paired with fifteen plausible but incorrect alternatives, producing a controlled ranking task measured by Hit@1.
Across seven open-weight models and one proprietary system, logit-based energy scoring yields a pooled Hit@1 of 33.0 percent, roughly double the 16.6 percent obtained by prompted listwise ranking. The single strongest configuration—a one-billion-parameter model using raw target-logit energy—reaches 53.1 percent, although this figure reflects the best of fourteen model-by-scorer combinations examined after the fact. The results indicate that a model’s unnormalized token-level can serve as a more stable signal for hypothesis validity than comparative prompting, at least within the constraints of the constructed benchmark.
The authors release both the curated dataset and the evaluation code, underscoring the importance of reproducibility for subsequent work on confidence-based methods in AI-assisted discovery.
Why it matters
Offers a novel, technically rigorous, and reproducible method for hypothesis evaluation that Dutch AI researchers can directly apply or extend in scientific-discovery pipelines; aligns with NL strengths in ethical/trustworthy AI and AI-for-science.
