AnchorBench: Benchmarking Anchoring Effect in LLMs

A new benchmark called AnchorBench has been developed by researchers to assess the anchoring effect in large language models (LLMs). This cognitive bias occurs when an initial reference point sways later evaluations. While the phenomenon is well-established in human behavior, recent investigations indicate that LLMs may also be affected. Previous studies, however, typically focused on a narrow range of anchor pathways and failed to differentiate between relevant and irrelevant anchors. AnchorBench remedies this by examining various anchor pathways along a defined relevance axis. The benchmark was evaluated on fourteen models, including ten open-weight and four frontier API models, using a comprehensive set of controlled prompts. Findings reveal that anchoring is significantly influenced by the pathway, with plausible anchors creating more substantial shifts than irrelevant ones, particularly through stronger pathways, and the impact diminishes as the anchor diverges from its original context. This research can be found on arXiv with the identifier 2608.14320.