LLMs Struggle with Implicature Cancellation in New Benchmark Dataset
A new study evaluates large language models' ability to recognize and update unspoken beliefs through implicature recognition and cancellation. The research introduces the first expert-annotated implicature cancellation dataset, crowdsourced for human judgments. Findings show LLM belief update understanding lags behind humans, especially in natural scenarios. Control experiments suggest successes may rely on prior beliefs, while failures depend on type and form.
Key facts
- Study evaluates LLMs on implicature recognition and cancellation
- First expert-annotated implicature cancellation dataset created
- Dataset crowdsourced for human judgments
- LLM belief update understanding lags behind humans
- Successes may stem from reliance on prior beliefs
- Failures depend on type and form of implicature
- Published on arXiv with ID 2607.25094
- Research focuses on pragmatic phenomenon of implicature cancellation
Entities
Institutions
- arXiv