whisper-small-hindi — openai/whisper-small fine-tuned for Hindi ASR on AI4Bharat Kathbath

Publisher: Zindagi Technologies Pvt Ltd, New Delhi · Model card version: 1.3 (2026-09-03) Public location: https://huggingface.co/zindagi-technologies/whisper-small-hindi (published 2026-09-03)

This is OpenAI's open-source whisper-small (244 M parameters, Apache-2.0) fully fine-tuned for Hindi speech-to-text on 134.75 hours of the AI4Bharat Kathbath Hindi corpus. On the held-out Kathbath Hindi test set the word error rate falls from 97.94% (base model, correctly configured for Hindi) to 14.87%; on the independent FLEURS hi_in test set from 80.23% to 19.54%. Every number below is derived from a file in the project repository; see Evaluation and Reproducibility.

Model details

Training data provenance (source, version, preprocessing)

Training data source. AI4Bharat Kathbath, Hindi configuration, train split, obtained from the Hugging Face dataset ai4bharat/Kathbath (gated, auto-approved on accepting the terms; licence cc-by-4.0 per the dataset card's metadata, visible after access is granted; the packaging is CC0). Kathbath is described in Javed et al. (2022), IndicSUPERB (arXiv:2208.11761): 1,684 hours of read speech across 12 Indian languages from 1,218 contributors in 203 districts, collected with a maker–checker verification flow; every sentence is spoken once, and the corpus provides a speaker-disjoint ("test-unknown") test split. We verified on the data actually used that no test speaker appears in training (below).

Version. Dataset fingerprint of the train split as served on 2026-09-02: 9638f862dc22c910 (valid: 623dd9832d844bba), recorded in logs/download_manifest.json. Rows: 91,752 train / 3,151 valid. Measured duration: train 134.75 h (mean clip 5.29 s, max 20.06 s), valid 4.84 h (logs/train_data_hours.json). Hindi speakers in the training split: 101.

Preprocessing (scripts/train.py, configs/train_small.json):

• audio decoded and resampled to 16 kHz mono; log-mel features (Whisper feature extractor, 80 mel bins, 3,000 frames) computed on the fly per batch, padded/truncated to 30 s;
• clips kept if 0.3 s ≤ duration ≤ 30 s and the tokenized transcript is < 448 tokens (Whisper's decoder position limit) — no clip was removed by these filters (91,752 → 91,752, logs/train/train_full.log);
• transcripts used verbatim as labels (no text normalization at training time); the Whisper tokenizer's language/task prefix set to <|hi|><|transcribe|><|notimestamps|>;
• shuffling with seed 42; a fixed 500-utterance subset of valid (seed 42) used for dev-WER.

Training procedure

Dev-WER (Vistaar-normalized) at steps 1,000 … 8,000 and final: 22.19, 17.78, 15.03, 14.70, 14.19, 13.39, 13.20, 12.96, 12.62.

Evaluation

Identical protocol for the base and fine-tuned model (scripts/eval.py): greedy decoding, language="hindi", task="transcribe", no timestamps, max_new_tokens=440, bf16. Headline WER/CER use the AI4Bharat/Vistaar text normalization (strip punctuation and danda marks → indic-nlp-library Hindi normalizer → collapse whitespace), applied identically to reference and hypothesis; raw (un-normalized) figures are given alongside.

Test-set integrity: the Kathbath test set is the official Vistaar benchmark archive (SHA-256 cb7d4fcdef4d3dc6…), cross-checked 100% against an independent mirror; a pre-training audit found 0 file-name overlaps and 2 verbatim transcript duplicates with the training split, which were excluded (n = 1,929 → 1,927); test and training speakers are disjoint (20 vs 101, overlap 0). FLEURS hi_in test: 418 recordings of 265 sentences from the official parquet (the dataset card's "920" is stale). The FLEURS runs are the corrected runs of 2026-09-03: a first version of the evaluation set had scored only 265 distinct recordings for the 418 rows because files were keyed on the FLEURS sentence id; the builder was fixed and both models re-evaluated (D2 report, §8).

Full report with per-utterance statistics, commands and the audit trail: D2 benchmark report (whitepaper/D2_BENCHMARK.md).

Intended use

• Short-form Hindi speech-to-text (≤ 30 s segments; longer audio chunked by the caller) of read speech in the style of the training data (news-like sentences read aloud). Related uses such as announcements, dictation or voice form-filling prompts are plausible but were not evaluated.
• As a starting point for further domain adaptation (customer calls, regional accents).
• Research and evaluation of Hindi ASR fine-tuning under a fully documented protocol.

Limitations and out-of-scope use

• Training data is read speech (134.75 h, 101 speakers). Conversational, telephone-quality, noisy or far-field audio was not in training and is not evaluated here; expect higher error.
• Not evaluated on Hindi–English code-switching. Kathbath transcribes English loanwords in Devanagari; the model does the same and did not emit Latin script on either test set.
• No long-form or timestamp capability was trained; the Whisper 30 s window applies. One FLEURS test clip (31.44 s) is truncated for both models.
• Numerals are the weakest category. The model writes numbers as words even where a reference uses digits; on the 37 Kathbath test sentences containing digits its WER is 48.31% (FLEURS: 65 sentences, 28.03%). Downstream systems needing digits must post-process.
• Speaker gender gap. Kathbath test WER is 11.51% on female-labelled and 18.34% on male-labelled recordings (n 971 / 956); no dialect or region breakdown is available.
• Decoding. Evaluation used greedy decoding with a 440-token cap (an earlier 225-token cap truncated long FLEURS utterances and was superseded — D2 §8.2). Use max_new_tokens up to 440 for dense 30 s segments. The base model's near-100% zero-shot WER is dominated by repetition loops; with beam search (5) it reaches 59.81% on Kathbath, still four times the fine-tuned model's 14.87% (D2 §8.3).
• The base model's zero-shot Hindi is poor (97.94% WER); the improvement figures should be read as adaptation of a general model to Hindi, not as a comparison with dedicated Hindi systems trained on 16–22× more data (see the whitepaper's positioning section).
• Single training run, single seed; bootstrap 95% CIs: Kathbath 14.17–15.52, FLEURS 18.39–20.77 (WER %).
• Two Kathbath test utterances were excluded for verbatim transcript overlap with the training split; results are on the remaining 1,927.
• No beam search was used for the fine-tuned model (a beam-5 baseline was run only as a sensitivity check); no language-model fusion or speaker adaptation was evaluated.
• Not for safety-critical transcription without human review.

How to use

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

repo = "zindagi-technologies/whisper-small-hindi"
processor = WhisperProcessor.from_pretrained(repo, language="hindi", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).eval()

# `audio` is a 16 kHz mono float32 numpy array of up to 30 s
features = processor.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
    ids = model.generate(input_features=features, max_new_tokens=440, num_beams=1)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Reproducibility

• Code, configuration, per-utterance evaluation outputs, training log and audit files are in the accompanying project repository (scripts/, configs/train_small.json, results/, logs/).
• The fine-tuned Kathbath evaluation was re-run from a rebuilt evaluation set on 2026-09-03, on the same machine and environment, and reproduced bit-identically (0 of 1,927 hypotheses differ), and again on a second host with a different driver under protocol v3 (0 of 1,927 differ). Throughput (RTX 4090, greedy, batch 16, bf16, excluding model load): about 39× real time on the second host and about 160× on the first (identical output; hosts differ).
• Exact package versions: logs/pod_env_freeze.txt (training host) and logs/pod/rerun/rerun_env_freeze.txt (protocol-v3 evaluation host); the scripts on both hosts were hash-verified against the committed versions (83088d3 and e0941aa).
• Weights checksum manifest: logs/d1_model_manifest.json.

Licences and attribution

• Weights: Apache-2.0, inherited from openai/whisper-small.
• Training data: AI4Bharat Kathbath — CC-BY-4.0 (dataset card, ai4bharat/Kathbath; packaging CC0; the Hub access terms were accepted by the publisher on 2026-09-02). Attribution: AI4Bharat, IIT Madras — Javed et al., IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages, arXiv:2208.11761 (2022).
• Evaluation data: Kathbath Hindi test (as above, via the Vistaar benchmark archive, Bhogale et al., Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR, Interspeech 2023, arXiv:2305.15386); Google FLEURS — CC-BY-4.0 (dataset card, google/fleurs; Conneau et al., FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, arXiv:2205.12446, 2022).
• Base model: Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, arXiv:2212.04356 (2022).
• Text normalization follows the AI4Bharat Vistaar evaluation recipe (MIT-licensed repository AI4Bharat/vistaar) using indic-nlp-library.

Citation

@misc{zindagi2026whispersmallhindi,
  title  = {whisper-small-hindi: openai/whisper-small fine-tuned for Hindi ASR on AI4Bharat Kathbath},
  author = {Zindagi Technologies Pvt Ltd},
  year   = {2026},
  note   = {Hugging Face model card, version 1.3}
}

Contact

Zindagi Technologies Pvt Ltd, New Delhi — https://zindagitech.com