First Dataset for LLM Legal Reasoning on Japanese Bar Exam Writing Task

Researchers have introduced the first dataset to evaluate large language models (LLMs) on open-ended legal reasoning within the Japanese jurisdiction, based on the writing component of the Japanese bar examination. The dataset requires LLMs to identify multiple legal issues from long narratives and construct structured legal arguments in free text. A key contribution is the manual evaluation of LLM responses by legal experts, revealing limitations and challenges in legal reasoning, including hallucinations. The study, published on arXiv (2604.23730), addresses a gap in prior research, which had focused on multiple-choice components of bar exams but lacked exploration of open-ended reasoning in realistic scenarios, particularly in the Japanese context.