IndiaFinBench: First LLM Benchmark for Indian Financial Text

What is IndiaFinBench

IndiaFinBench is the first publicly available evaluation benchmark specifically designed for assessing large language model performance on Indian financial regulatory text. Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety [1].

The benchmark addresses a significant coverage gap in existing financial NLP benchmarks, which draw exclusively from Western financial corpora including SEC filings, US earnings reports, and English-language financial news. IndiaFinBench provides a comprehensive evaluation framework tailored to the unique characteristics of Indian financial regulatory frameworks.

The benchmark contains 406 expert-annotated question-answer pairs sourced from 192 official documents from two primary Indian financial regulatory bodies: the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI).

What is new vs the previous version

IndiaFinBench represents the first benchmark of its kind, with no previous versions existing for Indian financial regulatory text evaluation.

How does IndiaFinBench work

IndiaFinBench operates through a structured evaluation framework that tests large language models across four distinct financial regulatory tasks.

1. Document Collection: Researchers gathered 192 official documents from SEBI and RBI regulatory publications
2. Question Generation: Expert annotators created 406 question-answer pairs distributed across four task categories
3. Task Distribution: Regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items)
4. Quality Validation: Annotation quality underwent dual validation through model-based secondary pass and human inter-annotator agreement evaluation
5. Zero-Shot Evaluation: Twelve models were tested under zero-shot conditions without task-specific training
6. Statistical Analysis: Bootstrap significance testing with 10,000 resamples identified statistically distinct performance tiers

Benchmarks and evidence

IndiaFinBench evaluation results demonstrate clear performance differentiation across twelve tested large language models.

Researchers and practitioners look at qualities like accuracy, efficiency, safety, fairness and robustness to determine how well a model performs [4]. The benchmark validation achieved strong inter-annotator agreement with kappa=0.611 and 76.7% overall agreement across 60 evaluated items.

Bootstrap significance testing with 10,000 resamples revealed three statistically distinct performance tiers among the evaluated models, providing robust statistical evidence for performance differentiation.

Who should care

Builders

AI researchers developing financial NLP systems need IndiaFinBench to evaluate model performance on non-Western regulatory frameworks. The benchmark provides standardized evaluation metrics for Indian financial text understanding capabilities.

Enterprise

Financial technology companies operating in Indian markets require IndiaFinBench to assess LLM suitability for regulatory compliance applications. Banks and fintech firms can use the benchmark to validate AI systems processing SEBI and RBI documentation.

End users

Financial professionals working with Indian regulatory documents benefit from IndiaFinBench-validated AI tools that demonstrate proven performance on regulatory interpretation and numerical reasoning tasks.

Investors

Venture capital and private equity investors evaluating AI companies serving Indian financial markets can reference IndiaFinBench results to assess technical capabilities and market readiness.

How to use IndiaFinBench today

IndiaFinBench is immediately accessible through its public GitHub repository with complete evaluation resources.

1. Access the Repository: Visit https://github.com/rajveerpall/IndiaFinBench to download the complete dataset
2. Download Components: Obtain the 406 question-answer pairs, evaluation code, and all model outputs
3. Set Up Evaluation: Install the provided evaluation framework following repository documentation
4. Run Baseline Tests: Execute zero-shot evaluation on your target LLM using the standardized protocol
5. Compare Results: Benchmark your model performance against the twelve reference models and human baseline
6. Analyze Task Performance: Examine results across the four task types to identify model strengths and weaknesses

IndiaFinBench vs competitors

IndiaFinBench stands alone as the first benchmark specifically designed for Indian financial regulatory text evaluation.

We evaluate PoliLegalLM on three representative benchmarks, including LawBench, LexEval, and a real-world dataset, PoliLegal [6], demonstrating the broader landscape of specialized domain benchmarks.

Risks, limits, and myths

• Limited Scope: IndiaFinBench focuses exclusively on SEBI and RBI documents, potentially missing other Indian financial regulatory bodies
• Zero-Shot Only: Current evaluation uses zero-shot conditions, which may not reflect fine-tuned model performance
• Language Limitation: The benchmark evaluates English-language regulatory text, excluding regional language financial documents
• Temporal Coverage: Document sources represent a specific time period, potentially missing recent regulatory changes
• Human Baseline: The 60.0% non-specialist human baseline may not represent expert-level human performance
• Task Balance: Uneven distribution across task types (174 regulatory interpretation vs 62 contradiction detection items)
• Model Selection: Evaluation limited to twelve models, potentially missing other relevant LLM architectures