LLM Reasoning Falls Short in Legal Case Forecasting on European Court of Human Rights

A new preprint on arXiv (2608.17168) investigates whether large language models (LLMs) can reason in a legally meaningful way when forecasting cases from the European Court of Human Rights (ECtHR). The study evaluates OpenAI GPT 5.4, a recent top-tier LLM, using alternative prompting strategies that vary in how much they suggest legally meaningful reasoning. Findings, assessed through both human and LLM evaluation, reveal that the model scores far from ideal in legal reasoning, producing structurally complete but substantively shallow analyses. LLM-as-a-Judge evaluators were internally consistent but aligned only weakly with trained annotators. The research underscores that despite reasoning capabilities in contemporary LLMs, their application to demanding legal tasks remains limited.