IndicIFEval is an evaluation dataset to assess the instruction-following capability of LLMs in Indic languages with verifiable, rule-based constraints.
Currently it supports 14 Indic languages, in addition to English.
| Assamese (as) | Nepali (ne) |
| Bengali (bn) | Odia (or) |
| Gujarati (gu) | Punjabi (pa) |
| Hindi (hi) | Sanskrit (sa) |
| Kannada (kn) | Tamil (ta) |
| Malayalam (ml) | Telugu (te) |
| Marathi (mr) | Urdu (ur) |
This repository contains the complete codebase for the IndicIFEval benchmark from Data Creation to Benchmark Evaluation.
This includes the scripts for indicifeval-trans creation, the pipeline for indicifeval-ground creation, and the custom evaluation configurations required for evaluation with lm-evaluation-harness.
Clone the repository and install the required dependencies. Please refer to the individual README for detailed usage.
The indicifeval-trans directory contains scripts to translate the English IFEval dataset into 14 Indic languages. Navigate to this directory and execute the main translation script to generate the localized prompts.
The indicifeval-ground directory houses the pipeline for synthetically generating instructions from native Indic content.
We use the Language Model Evaluation Harness for benchmarking. The lm-evaluation-harness directory contains the custom configurations required for our tasks. You must run the evaluation script specifying the model and the specific task configuration.
indicifeval-ground and indicifeval-trans do not use identical evaluation setups, even where a constraint category shares its name or checker code. In particular:
- Paragraph count + first word.
indicifeval-transtestsnum_paragraphsandnth_paragraph/first_wordtogether as independent, randomly-sized constraints. Inindicifeval-ground, this category was simplified to match how the underlying source content was collected:num_paragraphsis implicitly fixed at 1 and only the first-word constraint is evaluated. Releasedindicifeval-groundrows therefore omitnum_paragraphsfromkwargsfor this instruction — this is expected, not missing data. - Exact-count constraints (
keywords:frequency,length_constraints:number_sentences).indicifeval-groundprompts for these two constraint types are always phrased as an exact count (e.g. "the word X must appear exactly N times"), reflecting how the source text was mined (documents where a word/sentence count matched a target exactly). The evaluation harness now honors this: whenrelationis omitted inkwargsfor these two ground checkers, it defaults to an exact-equality comparison rather than randomly choosing "less than"/"at least".indicifeval-trans, by contrast, always supplies an explicitrelationand keeps the original less-than/at-least semantics from IFEval.
Not every translated prompt released in indicifeval-trans was judged correct by human review. Each row carries a tags list that includes exactly one of correct or incorrect (translation-quality verdict), and optionally parallel for the subset of rows that are aligned 1:1 across all 14 languages (used for the paper's cross-lingual comparison). To reproduce paper-reported numbers, filter to rows tagged correct (add parallel as well if you need the cross-lingual-aligned subset). Use for gold prompt or silver prompt for the indicifeval-ground release.
If you use IndicIFEval in your work, please cite us:
@article{jayakumar2026indicifeval,
title={IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages},
author={Thanmay Jayakumar and Mohammed Safi Ur Rahman Khan and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan},
year={2026},
eprint={2602.22125},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.22125},
}This dataset is released under the CC BY 4.0.