Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis
06:00 · August 17, 2026 · arXiv cs.AI RSS
Agentic "Continual Learning Harnesses", systems that pair an LLM with retrieval or memory to improve from feedback without retraining, have shown growing value in cybersecurity. But their value is conventionally measured by gains against labeled benchmarks, an approach that often fails in operational security settings. Benchmark labels are scarce, stale, and unrepresentative, so a practitioner often cannot tell whether a given harness helps at all or which of two is better for their task. Traditional LLM-as-a-judge offers little signal because it is no stronger than the agent it evaluates, and distillation is unreliable on scarce, sporadic, and biased labels. We propose a framework for evaluating learning harnesses end-to-end without a labeled benchmark, grounded in the scaling hypothesis. A stronger teacher model provides sparsely sampled corrections to a smaller student with a continual learning harness. We score a harness by how much its student converges toward the teacher over time. Across security tasks, model families, and harness designs, we show that improvement relative to the teacher correlates with improvement relative to a held-out gold standard, validating teacher-relative lift as a proxy for true harness uplift when labels are absent. We further show that LLM-as-a-judge between similarly powered models yields no usable signal. These results suggest that a teacher-sized model can be improved through the same harness when humans provide the same kind of sparse, high-precision corrections.
Summary
Agentic continual learning harnesses pair large language models with retrieval or memory mechanisms so that they can incorporate sparse human corrections over time without retraining. In operational cybersecurity these systems are attractive because threats evolve rapidly, yet their value is difficult to measure. Labeled benchmarks are typically scarce, quickly outdated, and biased toward the hardest cases, leaving practitioners without a reliable way to determine whether a given harness improves performance or to compare alternative designs.
The proposed evaluation framework rests on the scaling hypothesis. Two models from the same family are run over a large pool of unlabeled security-task examples; the larger model is designated teacher and the smaller one student wherever they disagree substantially. The student is then placed inside a continual learning harness and receives sparse corrections drawn from the teacher’s pseudo-labels. Harness quality is scored by the degree to which the student’s outputs converge toward the teacher on a held-out pseudo-labeled set. Experiments across multiple model families, security tasks, and harness architectures show that gains measured relative to the teacher correlate with gains measured against an oracle of human labels, supporting the use of teacher-relative improvement as a proxy when true labels are unavailable.
The same study demonstrates that an LLM-as-a-judge of comparable strength to the agent under test produces no usable signal, confirming that same-scale judges cannot reliably distinguish improvements above their own baseline. The results further indicate that a model of teacher capability could itself benefit from the harness when supplied with equally sparse, high-precision corrections from human analysts.
Why it matters
This research is highly relevant for Dutch AI researchers and cybersecurity practitioners facing data scarcity and privacy constraints. It offers a scientifically rigorous, label-free methodology to evaluate and improve AI agents in operational settings, aligning with the Netherlands' focus on robust, transparent, and applied AI solutions.
