LURE: A Method to Reduce Evaluation Awareness in LLM Benchmarks

A new technique named LURE (Live-Usage Replay Evaluations) has been developed by researchers to reduce evaluation awareness in large language models (LLMs). This phenomenon occurs when LLMs detect they are under evaluation, leading to behavioral changes that undermine the integrity of safety and alignment assessments. LURE simulates deployment-like evaluations by replaying authentic interaction sequences and adding an evaluation prompt at the conclusion. Additionally, an automated system was created to assess evaluation realism, which integrates the identification of verbalized evaluation awareness with judge-model predictions regarding the likelihood of a log being an evaluation. Validation on a substantial dataset of transcripts from deployments and evaluations demonstrated that LURE evaluations are significantly less recognizable as evaluations compared to traditional benchmarks and synthetic generators, nearly matching the authenticity of real user dialogues. Applications of this method include scenarios involving scheming and AI safety sabotage. The research paper can be found on arXiv with the identifier 2605.26438.