gemma4-e2b-vaani-asr-s2t-t2t

A full fine-tune of google/gemma-4-E2B-it for three tasks in one model: speech recognition, speech-to-text translation, and text-to-text translation, across English and dozens of Indian languages (the long tail of low-resource/tribal languages comes from Vaani, so ASR coverage goes well beyond the 19 languages listed in the metadata above — see Supported languages).

Averaged across all evaluated languages and directions, fine-tuning roughly halves ASR error (35% WER vs. 55% for the base model) and nearly doubles translation quality on both translation tasks (T2T: 47.6 vs. 25.0 chrF++; S2T: 36.4 vs. 18.7 chrF++) — see Results for the full per-language breakdown.

Layout

infer.py             inference (task, direction, input -- that's it)
prompt_contract.py   the exact prompts and trained language set, no dependencies
README.md
config.json, model.safetensors, tokenizer.json, ...   weights, config and tokenizer

Quick start

Same transformers usage as the base model (google/gemma-4-E2B-it) — just this repo's id, plus one of the three fixed prompts below.

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "ARTPARK-IISc/gemma4-e2b-vaani-asr-s2t-t2t"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto",
    attn_implementation="eager",  # matches training -- see below
)

ASR — transcribe speech, same language in and out:

messages = [{"role": "user", "content": [
    {"type": "audio", "audio": "clip.wav"},
    {"type": "text", "text": "Transcribe this audio in Hindi."},
]}]

inputs = processor.apply_chat_template(
    [messages], tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=256)
response = processor.tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True)
print(response)

S2T — translate speech into text in another language (same inputs / generate / decode as above, just a different prompt):

messages = [{"role": "user", "content": [
    {"type": "audio", "audio": "clip.wav"},
    {"type": "text", "text": "Translate this Hindi audio to English text."},
]}]

T2T — translate text (no audio part at all):

messages = [{"role": "user", "content": [
    {"type": "text", "text": "Translate the following text from Hindi to English.\n\nनमस्ते, आप कैसे हैं?"},
]}]

A ready-made infer.py/prompt_contract.py pair is also included, wrapping the above into an Infer class (load once, call .asr() / .s2t() / .t2t() repeatedly) plus a CLI:

from infer import Infer
m = Infer()
m.asr("clip.wav", lang="Hindi")

python infer.py --task asr --src-lang Hindi --audio clip.wav

The prompts are part of the model

Hardcoded in prompt_contract.py, verbatim from training — these are not configurable at training time, and rewording them was not tested, so don't expect a paraphrase of these to work as well. infer.py always goes through this file's build_prompt_* functions rather than formatting the instruction itself:

ASR:  Transcribe this audio in {lang}.
S2T:  Translate this {src} audio to {tgt} text.
T2T:  Translate the following text from {src} to {tgt}.\n\n{text}

Wrapped in Gemma 4's chat template (<|turn>user ... <turn|>, not Gemma 2/3's <start_of_turn>) — use processor.apply_chat_template, don't hand-write turn markers.

Supported languages

ASR

English, Assamese, Bengali, Gujarati, Hindi, Kannada, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Tamil, Telugu, Urdu, Bhojpuri, Chhattisgarhi, Garo, Magadhi, Mizo — plus ~50 additional lower-resource and tribal languages from Vaani (Angika, Awadhi, Garhwali, Khasi, and many more); accuracy on the long tail varies widely with how much training data each one had — see eval results for the pattern.

T2T

English ↔ Assamese, Bengali, Bodo, Dogri, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, Urdu (22 languages). Both Indic→English and English→Indic are supported; Indic↔Indic is not (see Limitations).

S2T

English ↔ Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, Urdu (16 languages). Both Indic→English and English→Indic are supported, though coverage is thin for Sanskrit and Sindhi (see Results); Indic↔Indic is not (see Limitations).

Results

Scored on held-out test sets built specifically for this evaluation (not seen during training). ASR scored with WER (bracket/punctuation-normalized, applied to both reference and hypothesis, and scored against code-switched references where available — e.g. "mobile" accepted in place of "मोबाइल" — since some sources' native-script references include loanwords the model correctly renders in Latin script, which naive native-script WER penalizes as an error); S2T/T2T scored with chrF++.

T2T — IN22-Gen (chrF++, higher better)

S2T — FLEURS (chrF++, higher better)

ASR — FLEURS + KathBath + Vaani (WER%, lower better)

Limitations

• English-pivot only, same as the text-only translation sibling model. Every S2T/T2T training pair had English on one side; Indic↔Indic translation is untrained. Earlier probing found this model generalizes to unseen Indic↔Indic text pairs reasonably well but not to unseen Indic↔Indic speech pairs — an architectural/training-dynamics asymmetry between the two pathways, not a data-coverage gap (neither was ever trained on direct Indic↔Indic pairs).
• Long low-resource tail is weak. Many Vaani-only languages score 80-150%+ WER — a data-scarcity problem, not something more training fixes.
• Noisy/field-recorded audio is harder than clean studio audio for the same language, by a consistent ~2x WER gap (Vaani vs. FLEURS/KathBath).
• Audio capped at 30 seconds per clip.
• Unused modality. The checkpoint keeps Gemma-4's vision tower, frozen throughout and untouched by fine-tuning — dead weight for these three tasks, kept so the checkpoint loads with the stock model class.

License

Derived from google/gemma-4-E2B-it and therefore governed by the Gemma Terms of Use; the prohibited-use policy applies to this model as it does to the base. Training data mixes BhasaAnuvaad, BPCC, and local Vaani-sourced ASR data from AI4Bharat/IISc — check each dataset's own licence terms for your use case.

Acknowledgements

Trained with compute awarded by the Gemma Academic Program, supported by google.org. We thank Google for the award.