IndiaFinBench: First LLM Benchmark for Indian Financial Regulation

What is IndiaFinBench

IndiaFinBench is the first publicly available evaluation benchmark specifically designed to assess large language model performance on Indian financial regulatory text. The benchmark addresses a critical gap in existing financial NLP evaluation tools, which have historically drawn exclusively from Western financial corpora including SEC filings, US earnings reports, and English-language financial news.

The benchmark contains 406 expert-annotated question-answer pairs sourced from 192 official documents from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI). These regulatory documents represent the primary financial oversight bodies in India and cover diverse aspects of financial regulation, compliance requirements, and policy frameworks.

What is new vs the previous version

IndiaFinBench represents an entirely new benchmark category, as no previous evaluation framework has focused on Indian financial regulatory text.

How does IndiaFinBench work

IndiaFinBench operates through a structured four-task evaluation framework designed to assess different aspects of regulatory text comprehension.

1. Regulatory Interpretation (174 items): Models must demonstrate understanding of complex regulatory language and policy implications from SEBI and RBI documents.
2. Numerical Reasoning (92 items): Tasks require mathematical computation and quantitative analysis of financial regulations and compliance requirements.
3. Contradiction Detection (62 items): Models identify inconsistencies or conflicting statements within regulatory text passages.
4. Temporal Reasoning (78 items): Evaluation focuses on understanding time-dependent regulatory changes and chronological policy relationships.

The benchmark employs zero-shot evaluation conditions, meaning models receive no task-specific training examples before assessment. Annotation quality validation includes both model-based secondary passes and human inter-annotator agreement evaluation across 60 items.

Benchmarks and evidence

IndiaFinBench evaluation results demonstrate significant performance variation across twelve tested models under zero-shot conditions.

Bootstrap significance testing with 10,000 resamples revealed three statistically distinct performance tiers among evaluated models. The numerical reasoning task category showed the highest discrimination between model capabilities, with performance gaps exceeding 35 percentage points between top and bottom performers.

Who should care

Builders

AI developers working on financial applications can use IndiaFinBench to evaluate model performance on regulatory compliance tasks. The benchmark provides standardized metrics for assessing LLM capabilities in non-Western financial contexts, enabling more robust model selection and fine-tuning strategies.

Enterprise

Financial institutions operating in India can leverage IndiaFinBench to assess AI systems for regulatory compliance automation. Banks, investment firms, and fintech companies can evaluate whether their LLM implementations meet accuracy requirements for processing SEBI and RBI documentation.

End users

Researchers and academics studying financial NLP can access a comprehensive evaluation framework for Indian regulatory text. The benchmark enables comparative analysis of model performance across different regulatory interpretation tasks.

Investors

Investment firms can use IndiaFinBench results to evaluate AI-powered compliance and regulatory analysis tools. The benchmark provides objective performance metrics for assessing fintech solutions targeting Indian financial markets.

How to use IndiaFinBench today

IndiaFinBench is immediately accessible through its GitHub repository for research and evaluation purposes.

1. Access the repository: Navigate to https://github.com/rajveerpall/IndiaFinBench to download the complete dataset and evaluation code.
2. Load the benchmark data: The repository contains 406 question-answer pairs organized by task type with corresponding SEBI and RBI source documents.
3. Implement evaluation protocol: Use the provided evaluation code to assess your model under zero-shot conditions across all four task categories.
4. Compare results: Benchmark your model performance against the twelve baseline models with accuracy scores ranging from 70.4% to 89.7%.
5. Analyze task-specific performance: Focus on numerical reasoning tasks for the most discriminative evaluation of model capabilities.

IndiaFinBench vs competitors

IndiaFinBench occupies a unique position in the financial NLP benchmark landscape by focusing specifically on Indian regulatory frameworks.

Risks, limits, and myths

• Geographic limitation: The benchmark focuses exclusively on Indian regulatory frameworks and may not generalize to other financial jurisdictions.
• Language constraint: All evaluation materials are in English, potentially missing regional language regulatory documents used in Indian financial contexts.
• Temporal scope: The benchmark reflects regulatory frameworks as of the document collection date and may not capture recent policy changes.
• Task coverage: Four task types may not encompass all aspects of regulatory text comprehension required in real-world financial applications.
• Human baseline limitation: The 60.0% non-specialist human baseline may not represent expert-level human performance on these tasks.
• Model selection bias: Evaluation limited to twelve models may not represent the full spectrum of available LLM capabilities.