New Framework Evaluates Agentic Learning Harnesses Without Labels

A recent preprint on arXiv (2608.13608) introduces a novel framework for assessing agentic continual learning harnesses that does not depend on labeled benchmarks. These harnesses, which integrate a large language model (LLM) with retrieval or memory to learn from feedback without the need for retraining, are becoming increasingly important in cybersecurity. Traditional evaluation techniques that utilize labeled benchmarks often fall short in operational security contexts due to outdated, limited, and unrepresentative labels. Additionally, LLM-as-a-judge methods yield minimal insights, as they are no more effective than the evaluated agent, and distillation methods struggle with sparse and biased labels. The new framework is based on the scaling hypothesis, where a robust teacher model offers infrequent corrections to a smaller student with a continual learning harness, which is evaluated based on its enhancement of the student's performance. This methodology seeks to establish a more effective and realistic evaluation system for security applications in the real world.