LLMs Show Recency Bias in Temporal Legal Reasoning, Study Finds
A recent investigation published on arXiv (paper 2608.14610) explores how large language models (LLMs) assess which legal version is relevant to a specific case, a process known as 'temporal applicable-law determination.' The study establishes a benchmark for evaluating LLM performance in this area and analyzes their shortcomings. Four primary conclusions are drawn: firstly, LLMs tend to favor the most recently passed law, irrespective of when the pertinent facts occurred. Secondly, this tendency is not attributed to a lack of understanding of the temporal nature of laws or ignorance of past statutes. Thirdly, evidence indicates that reinforcement-learning-driven explicit reasoning might inadvertently strengthen this recency bias. The research highlights significant limitations for legal judgment prediction (LJP) and other AI applications in law, emphasizing the importance of temporal precision. The paper can be accessed on arXiv with the identifier 2608.14610.
Key facts
- Study from arXiv paper 2608.14610
- Focuses on temporal applicable-law determination in legal reasoning
- Constructs a benchmark to evaluate LLMs on this task
- Finds LLMs bias toward most recently enacted law
- Bias not due to lack of understanding of temporal scope or historical statutes
- Reinforcement-learning-shaped explicit reasoning may be a key mechanism
- Implications for legal judgment prediction (LJP) and legal AI
- Paper available on arXiv
Entities
Institutions
- arXiv