Measuring Cross-Task Behavioral Consistency in Language Model Agents

A recent paper on arXiv presents the Behavioral Consistency Metric (BCM), aimed at measuring the consistency of behavior in language model agents. The researchers contend that conventional metrics, such as success rate, do not adequately reflect an agent's consistency across various tasks. BCM involves training a model to forecast task success based on behavioral features from agent execution traces, generating feature-attribution vectors for each trajectory, and calculating the average pairwise similarity of these vectors within an agent system. By examining approximately 9,000 trajectories from six language model agents engaged in software engineering tasks, the findings reveal that cross-task and within-task consistency can differ: some systems exhibit local reproducibility but lack global coherence, while others maintain consistency at both levels. The paper can be found on arXiv with the identifier 2608.13598.