LLM Judges for Recommendation Evaluation: A New Alignment Framework
A recent study published on arXiv (2608.11493) explores the application of Large Language Models (LLMs) as evaluators for offline recommendation systems, uncovering a significant flaw known as 'bidirectional rationalization.' In zero-shot scenarios, LLMs can persuasively present arguments for both favorable and unfavorable user interactions regarding the same item using the same evidence, which raises concerns about their dependability. To mitigate this issue, the researchers introduce a sequential behavioral alignment framework that combines fine-tuning with preference optimization based on paired correct and counterfactual rationales. When tested on actual homepage interaction data, this method results in a 32.19% improvement in Macro-F1 score compared to the zero-shot baseline. The findings highlight LLMs' potential in recommendation evaluation while stressing the importance of alignment for reliability.
Key facts
- The study is from arXiv:2608.11493.
- Traditional offline recommendation evaluation relies on complex, manually maintained feature pipelines.
- LLMs offer a promising alternative by predicting user engagement from raw text logs.
- The identified failure mode is 'bidirectional rationalization'.
- In zero-shot settings, LLMs argue for both positive and negative outcomes on the same item.
- The proposed framework is sequential behavioral alignment with fine-tuning and preference optimization.
- Evaluation used real-world homepage interaction logs.
- The aligned approach achieved a 32.19% lift in Macro-F1 score over zero-shot baseline.
Entities
Institutions
- arXiv