Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?
06:00 · June 24, 2026 · arXiv cs.AI RSS
Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can assist with this explanation problem once a circuit has already been identified. We introduce AgenticInterpBench, a benchmark for circuit explanation built from 84 semi-synthetic transformer circuits with 163 component-level annotations. We propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that analyzes each component through an iterative loop of observation, hypothesis generation, and causal validation, eventually producing a component-level explanation and a circuit-level task description. Across four LM backbones, HyVE recovers useful component- and task-level explanations, but no backbone is uniformly best. Our analysis shows that strong backbones usually form observation-grounded hypotheses, while failures more often arise later in the validation loop, through incomplete validation plans, code execution errors, or unresolved hypotheses. A case study on an arithmetic circuit in Llama-3-8B shows that the same formulation can extend beyond semi-synthetic benchmarks to naturally trained models. Overall, LM agents are promising circuit explainers, but reliable validation remains the key obstacle.
Summary
Mechanistic interpretability has automated much of the work of localizing circuits—sub-networks of attention heads and MLP layers that implement specific behaviors in transformer models—yet turning those localized components into coherent functional explanations remains a manual, hard-to-scale process. Researchers typically iterate through observation, hypothesis formation, and causal tests, but the volume and complexity of modern models make this approach increasingly impractical.
To examine whether language-model agents can take on this explanation task, the authors created AgenticInterpBench. The benchmark contains 84 semi-synthetic circuits drawn from InterpBench, covering 163 annotated components. Each component carries a functional-role tag from a five-class taxonomy plus a natural-language description of its task-specific behavior; the circuits also come with input-output examples that illustrate the overall computation.
The proposed agent framework, HyVE, processes each circuit through repeated cycles of observation, hypothesis generation, and causal validation. It ultimately outputs both component-level explanations and a circuit-level task description. When tested with four different language-model backbones, HyVE reached component-tag accuracy as high as 79 percent and task-description accuracy as high as 83 percent. Stronger backbones tended to produce observation-grounded hypotheses, whereas most failures occurred later in the loop through incomplete validation plans, execution errors, or unresolved hypotheses.
A separate case study applied the same agent to an arithmetic circuit previously identified in the naturally trained Llama-3-8B model. The agent recovered the main operand-transfer structure and correctly or partially correctly described most components, indicating that the approach can transfer beyond the semi-synthetic setting. Across all experiments, the dominant remaining obstacle is reliable causal validation rather than initial hypothesis formation.
Why it matters
This research is highly relevant to the Dutch and EU focus on ethical, transparent, and explainable AI. Advancements in mechanistic interpretability provide researchers and practitioners with the tools needed to understand complex models, directly supporting compliance with the transparency requirements of the EU AI Act.
