HumRightsBench: First Benchmark for LLM Reasoning in Human Rights Law

A new pilot methodology aims to create the first expert-validated, scenario-based benchmark for evaluating large language models (LLMs) on their ability to reason about international human rights law. The benchmark, named HumRightsBench, addresses the growing role of LLMs in mediating legal determinations that affect human rights. The methodology adapts the IRAC framework—traditionally used in legal reasoning—by substituting the 'conclusion' step with 'proposing remedies,' resulting in an IRAP structure tailored to human rights work. The pilot series includes authentic scenarios annotated by human rights lawyers and professionals worldwide, covering various dimensions of real-world human rights issues. The research is reported in a paper on arXiv (ID: 2608.10268), which details the development of a robust and scalable approach. The findings indicate that current models face challenges in this domain, underscoring the need for such benchmarks. This initiative represents a significant step toward ensuring that LLMs can correctly apply human rights law in practical contexts.