Is it agentic enough? Benchmarking open models on your own tooling

Summary

This article presents a benchmarking harness that evaluates coding agents not only by whether they reach a correct answer when using a library such as Hugging Face Transformers, but also by the effort required along the way. The harness records token consumption, latency, number of turns, and the specific API paths or files an agent consults, then compares these quantities across model sizes, library revisions, and three distinct tiers of assistance: a bare environment, a cloned repository, and a curated skill package that supplies task-specific documentation and a command-line interface.

The work treats Transformers as a representative case study. Agents are given typical machine-learning tasks such as text classification, image captioning, and audio transcription, and are driven by open models ranging from large frontier-scale systems to smaller local ones. Because large models usually reach a correct result, the primary signals become the cost of reaching it; for smaller models, success rate itself remains informative. The harness therefore surfaces different metrics for each regime and records full agent traces so that library maintainers can inspect exactly which commands or files were used.

Results show that introducing a dedicated CLI together with structured documentation and examples reduces median time and turns for the largest models, largely because they adopt the new interface instead of debugging Python calls. The same commit, however, increases token usage in the clone tier, as agents spend additional input tokens reading the newly added CLI implementation and example scripts. Smaller models exhibit the opposite pattern: the added surface can lower match rates, apparently because they rely more on memorized pipeline patterns and are more easily distracted by unfamiliar constructs.

The authors therefore argue that any change intended to improve agent interaction must be measured across the full range of model capabilities rather than validated only on the strongest systems. The released harness and its client-side report allow maintainers to run such experiments on their own repositories and to inspect both aggregate statistics and individual traces.

Why it matters

It provides ML Engineers with actionable insights and a new open-source tool to benchmark and optimize their own libraries for agentic use. Understanding the trade-offs in token consumption and latency across different model sizes is crucial for building cost-effective and reliable AI systems.