Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering
06:00 · July 1, 2026 · arXiv cs.AI RSS
ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.
Summary
HASTE addresses the repeated cold-start problem in ML engineering agents, where each Kaggle competition is treated independently and prior successful techniques must be rediscovered from scratch. The system maintains a growing inventory of skills organized into three explicit scope tiers: global skills applicable across ML tasks, domain skills specific to tabular, vision, NLP or audio data, and competition-specific skills tied to a single dataset. An orchestrator routes work to matching domain specialists and uses LLM-driven abstraction to promote useful patterns upward between tiers, ensuring that only relevant knowledge is loaded into each agent’s context.
Controlled experiments holding a 159-skill inventory fixed across eight competitions show that tiered loading produces a 100 % medal rate, while loading the same skills as a flat pool yields only 62.5 %—identical to loading no skills at all—and doubles output-token consumption. On the full MLE-Bench Lite suite of 22 competitions, HASTE reaches a 77.3 % medal rate using Claude Sonnet 4.6 with a 12-hour budget per task. Warm-start runs that reload previously accumulated global and domain skills require 52 % fewer refinement iterations and raise the fraction of accepted code changes from 42 % to 85 % once more than 50 skills are available.
These results indicate that explicit scope-based organization of accumulated knowledge can materially reduce redundant exploration, allowing a non-frontier model under a modest compute budget to match performance levels previously associated with stronger models or longer runtimes. The authors note that the reported medal rate is a single-seed outcome and identify multi-seed replication as the next verification step.
Why it matters
This research is highly relevant for Dutch AI researchers and practitioners as it offers a concrete methodology to reduce compute costs and improve the efficiency of AI development through transfer learning in multi-agent systems. Its focus on resource efficiency aligns well with the Dutch AI market's emphasis on sustainable and scalable AI solutions for enterprises and SMEs.
