Measuring Cross-Task Behavioral Consistency in Language Model Agents
06:00 · August 17, 2026 · arXiv cs.AI RSS
Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves. We argue that behavioral consistency across tasks is a distinct and measurable property, and we introduce the Behavioral Consistency Metric (BCM) to quantify it. BCM trains a model to predict task success from behavioral features of agent execution traces, derives a per-trajectory feature-attribution vector, and measures the mean pairwise similarity of these vectors within an agent system. Across roughly 9,000 trajectories from six language model agents on software engineering tasks, our central finding is that cross-task and within-task consistency are distinct axes that can diverge: some systems are locally reproducible, behaving similarly on repeated attempts at one task, yet globally fragmented, with no stable strategy across different tasks, while others are consistent at both scales. Prior work measures only same-task reproducibility and so cannot observe this separation. We further find that consistency is not reducible to success rate, since systems with comparable success can differ sharply in consistency, and that the frontier-versus-open-source consistency gap persists under a within-task control that holds task difficulty constant. We position BCM as a process-level reliability signal that complements outcome metrics, and we are explicit about the conditions under which it is meaningful.
Summary
Agent evaluation has long centered on outcome metrics such as success rate, which register whether a language model agent completes a task but remain silent on the stability of its underlying behavior. The paper introduces the Behavioral Consistency Metric (BCM) to address this gap by quantifying how consistently an agent applies recognizable strategies across different tasks. BCM extracts a fixed set of structural features from each execution trajectory—covering step count, error frequency, action diversity, and the balance between navigation, viewing, and editing—then trains a predictor to map these features to task success. For every trajectory the method produces an attribution vector that records the contribution of each feature to the success prediction; BCM is simply the average pairwise similarity among these vectors for a given agent system.
Experiments on roughly 9,000 trajectories from six agents operating on software-engineering benchmarks show that cross-task consistency and within-task reproducibility are separable dimensions. Some agents reproduce similar attribution patterns on repeated attempts at one task yet display markedly different patterns when moving to new tasks, indicating local reproducibility without a stable global strategy. Others maintain coherent representations at both scales. The study further demonstrates that consistency cannot be inferred from success rate alone: agents achieving comparable outcome scores can differ substantially in the similarity of their attribution vectors. The performance gap between frontier and open-source models persists even after controlling for task difficulty within individual tasks.
By treating attribution vectors as compact behavioral signatures, BCM supplies a process-level signal that complements existing outcome measures and clarifies when an agent’s behavior can be characterized, anticipated, and trusted within known bounds.
Why it matters
The article provides a novel, quantifiable method for assessing the reliability and behavioral consistency of AI agents, which is crucial for compliance with EU AI regulations and the Dutch focus on transparent AI. Researchers can directly apply the open-source BCM framework to evaluate and improve the predictability of enterprise AI deployments.
