Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1
06:00 · July 9, 2026 · arXiv cs.AI RSS
Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures. We study a third regime: an open-weight model in non-thinking mode (DeepSeek V3.2) under a strict budget, with no ARC-specific fine-tuning. We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly. First, we introduce an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis, implemented as a two-stage agent pipeline. Next, we present the Reflective Orchestrator, which augments the pipeline with autonomous exploration of new transformations when previous hypotheses fail on training pairs. On the ARC-AGI-1 public 400-task evaluation set, the pipeline reaches 57.50% pass@2 at \$0.25 per task, and the orchestrator reaches 67.25% pass@2 at \$0.62 per task. Together these architectures lift a 15.50% one-shot baseline by ~52 points without benchmark-specific training or heavy test-time compute. Furthermore, the orchestrator-driven lift tests a falsifiable diagnostic the pipeline produces; unbiased pass@k analysis suggests the pipeline is generation-bound, not selection-bound (selection via training-pair accuracy captures ~95% of the candidate ceiling) and predicts that significant improvement requires broader generation, not better ranking. The orchestrator implements this prediction via adaptive re-exploration and confirms it (unbiased pass@1 lift +9.81 pp, matching selection-mediated pass@2 lift). An additional pipeline ablation identifies its think tool as a significant component, with removal reducing pass@2 by 5.75 pp.
Summary
Recent progress on the ARC-AGI-1 benchmark has largely relied on two resource-intensive strategies: either heavy test-time compute applied to frontier models through evolutionary search or extended chain-of-thought, or benchmark-specific training that fine-tunes smaller models directly on ARC data. This work instead examines a third regime that uses an open-weight model, DeepSeek V3.2, in non-thinking mode under a strict per-task budget and without any ARC-specific fine-tuning or reinforcement learning on the benchmark.
The authors introduce two agentic architectures that decompose the problem into explicit stages of pattern discovery and program synthesis. The Explorer-Definer Pipeline separates these concerns into a two-stage process: an explorer stage identifies candidate patterns from input-output pairs, while a definer stage synthesizes executable transformations that are verified against the training examples. On the public 400-task evaluation set this pipeline reaches 57.50% pass@2 at an average cost of $0.25 per task.
The Reflective Orchestrator augments the pipeline with an adaptive mechanism that triggers fresh exploration of transformations whenever a candidate fails on the training pairs. This addition lifts performance to 67.25% pass@2 at $0.62 per task. The gain is shown to be generation-driven rather than selection-driven: unbiased pass@k analysis indicates that training-pair accuracy already captures roughly 95% of the candidate ceiling, so further improvement requires broader hypothesis generation. An ablation confirms that a think scratchpad within the definer contributes an additional 5.75 percentage points.
Together the architectures raise a 15.50% one-shot baseline by more than 50 points while remaining well below the cost of prior high-performing systems. The results are obtained on the public evaluation split; the authors note that direct comparison with official ARC Prize leaderboard scores is not possible without evaluation on the semi-private holdout set.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises looking to deploy advanced reasoning capabilities cost-effectively. Its focus on open-weight models and architectural efficiency aligns with the Netherlands' push for sustainable, accessible, and transparent AI solutions without relying on massive compute budgets.
