Model Routing Is Simple. Until It Isn’t.

Summary

Model routing in agentic systems is often framed as a classification task that assigns each request to an appropriate model. In practice the problem expands into multi-objective systems optimization once cost, accuracy, latency, compliance, and infrastructure state must be balanced simultaneously. Benchmarks run on 417 AppWorld tasks with a CodeAct agent illustrate how surface-level signals quickly become unreliable.

Observed costs frequently diverge from published token prices because of caching. In the AppWorld evaluation, Claude Sonnet 4.6 produced a total bill of $79 while GPT-4.1 reached $155, even though Sonnet generated roughly three times as many reasoning steps. The difference arose from Sonnet’s lower cache-read pricing and the high reuse of context typical in agent workloads; routers that consult only price sheets therefore optimize against incorrect values.

Task difficulty is another unreliable signal. A request that appears straightforward may trigger retrieval, tool calls, or repeated refinement, while a technically dense prompt can be resolved efficiently by a smaller model. Production routers must also respect data-residency rules, approved-model lists, and privacy constraints that can override any cost or quality preference.

Latency likewise depends on more than model size. Routing decisions themselves add overhead, and factors such as hardware placement, cache warmth, and endpoint load often dominate end-to-end response time. Routing at every step increases flexibility but multiplies both latency and operational complexity.

An optimization-based router addresses these issues by tracing cost-accuracy frontiers rather than selecting a single “best” model. On the same AppWorld suite, lightweight configurations achieved 84 percent accuracy at $93 total cost and 83 seconds of latency, delivering a 21 percent cost reduction and 9 percent latency reduction relative to Opus alone for a 4 percent accuracy trade-off. The router itself required roughly 6 ms and 2 kB of memory per task, ensuring it did not become a bottleneck. The central insight is that effective routing optimizes operating points across the entire system rather than performing isolated model classification.

Why it matters

Directly actionable for ML Engineers building production routers: covers latency/VRAM-adjacent serving realities, cost-accuracy tradeoffs, and EU-relevant compliance/data residency rules. Provides concrete metrics and an optimization approach applicable to Dutch SME and enterprise deployments.