IndiaFinBench: First LLM Benchmark for Indian Financial Regulatory Text
Researchers have introduced IndiaFinBench, the first open benchmark designed to assess how well large language models (LLMs) perform on Indian financial regulatory documents. This new benchmark addresses a major gap, as most existing financial NLP evaluations rely on Western sources like SEC filings and US earnings reports. IndiaFinBench includes 406 question-answer pairs, meticulously annotated, sourced from 192 documents from the Reserve Bank of India (RBI) and the Securities and Exchange Board of India (SEBI). It features four task categories: regulatory interpretation (174), numerical reasoning (92), contradiction detection (62), and temporal reasoning (78). The annotation quality is validated with a model-based secondary check, showing a kappa of 0.918 for contradiction detection and a human agreement rate on 60 items.
Key facts
- IndiaFinBench is the first publicly available evaluation benchmark for LLM performance on Indian financial regulatory text.
- Existing financial NLP benchmarks draw exclusively from Western financial corpora.
- The benchmark includes 406 expert-annotated question-answer pairs.
- The pairs are drawn from 192 documents from SEBI and RBI.
- Four task types are covered: regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning.
- Regulatory interpretation has 174 items, numerical reasoning 92, contradiction detection 62, and temporal reasoning 78.
- Annotation quality is validated with a model-based secondary pass (kappa=0.918 on contradiction detection).
- A 60-item human inter-annotator agreement evaluation was also conducted.
Entities
Institutions
- Securities and Exchange Board of India (SEBI)
- Reserve Bank of India (RBI)
Locations
- India