Bhasha v2: word-level language ID for code-mixed Indian-language text

Bhasha v2 (bhasha-lid-indic-v2) tags every word of a message with its language: Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia or English, plus name and other. It reads Roman letters ("padam semma mass"), native scripts, and messages that mix both. 278M-parameter XLM-RoBERTa-base token classifier, Apache-2.0.

📖 Write-up: My Model Thought Tamil Was Telugu (Towards AI) · Summary and FAQ · By Arun Prabhakaran · @LatentAILabs

What changed from v1: v1 learned only news-style text converted to Roman letters, and called about a third of spoken Tamil words Telugu. v2 adds 51,071 real YouTube comments in Tamil, Malayalam and Kannada (DravidianCodeMix, CC-BY-4.0), word-labelled by an open-weights model (Gemma 4 26B, Apache-2.0). Spoken Tamil goes from 0.72 to 0.94 on human-labelled comments; every other independent test set improves, except native-script and romanised single-language sentences, which stay within 0.005.

Results

Word-level macro-F1 over {Indic language, English}; name, other and mixed words (an English stem with an Indic ending) are left out of scoring. v1 and v2 were each scored once with the frozen model. None of these sets was used for training.

Independent human-labelled sets (these decided whether v2 ships):

The CoLI files have one word per row and no sentence boundaries, so they were scored in contiguous 12-word windows. Part of CoLI Malayalam (and a little of CoLI Tamil) overlaps v2's new DravidianCodeMix training comments. On only the windows with no 5-word overlap, the scores are the same: Tamil v1 .718 -> v2 .935 (1,219 windows), Malayalam v1 .912 -> v2 .967 (939 windows). Kannada, Kanglish and WoLLaI have no overlap. Only 41% of CoLI Malayalam windows are clean, so it is not counted as independent for v2; the ship rule also passes without it. distilka still beats v2 on CoLI Kannada, a test from the same shared-task series as its training data; v2 leads on the other Kannada sets. distilka and the Malayalam model state no licence.

Other checks (reported, not independent for v2):

Known weaknesses

• Telugu, Marathi, Punjabi, Urdu and Odia have no real code-mixed test; only native-script and romanised single-language sentences were tested. v2's new real text is Tamil, Malayalam and Kannada only. On romanised Telugu sentences v2 is unchanged (F1 .925 -> .926; 1 of 300 called Tamil), but v2 saw no Telugu comments, so casual code-mixed Telugu may now be called Tamil more often; this is untested.
• Hindi words inside Gujarati–Hindi–English mixing are weak (F1 .32).
• Whole romanised sentences in one language: GlotLID is better (.994 vs .903).
• Labels of the new training comments were written by an open-weights model, checked at 7.5% word error against 150 comments labelled by three LLM labellers (not human-verified).
• Hindi vs Urdu in Roman letters often cannot be told apart; the wrapper returns hi/ur when unsure.
• One- or two-word messages carry little context.

Use

# pip install transformers torch huggingface_hub
import os, sys
from huggingface_hub import hf_hub_download
sys.path.insert(0, os.path.dirname(hf_hub_download("LatentAILabs/bhasha-lid-indic-v2", "lid_wrapper.py")))
from lid_wrapper import LID

lid = LID("LatentAILabs/bhasha-lid-indic-v2")
lid("ನಾಳೆ meeting ಇದೆ")                       # {"words": [(word, label), ...], "main": ..., "script": ..., "mixed": ..., "share": {...}}
lid("bro naan nalaiku office varala", langs=["ta"])   # optional: limit output to the expected language + English
lid.tag(["kal", "meeting", "hai"])                    # list of words -> list of labels

Labels: hi bn ta te mr gu kn ml pa ur or en name other, plus hi/ur from the wrapper. The raw model is a standard XLMRobertaForTokenClassification; label the first sub-token of each word.

Training data

• v1's 1.65M code-built rows: native sentences from IndicCorp v2 (CC0), romanised with Aksharantar (CC-BY 4.0), the IndicXlit model (MIT) and rules weighted by FineWeb-2 word counts (ODC-By), mixed with English, plus names from Wikidata (CC0) and chat-style noise.
• New in v2: 51,071 real YouTube comments (Tamil 34,633, Malayalam 10,975, Kannada 5,463) from the DravidianCodeMix offenseval training split (CC-BY-4.0), used twice. Word labels by Gemma 4 26B-A4B-it (Apache-2.0) following a written labelling guide, temperature 0. Rows overlapping any evaluation set were removed; 300 + 150 comments were held out for checking. A simpler rule-based labeller was tried first and rejected (11.6% word error).
• XLM-RoBERTa-base, 1 epoch, lr 3e-5, batch 64, one L4 GPU (about an hour); checkpoint chosen on 300 held-out comments.
• No proprietary-model output is in the training data.

Evaluation data credits

CoLI-Dravidian 2024 (FIRE 2024 shared task) and CoLI-Kanglish (ICON 2022) were used for evaluation only and are not redistributed. WoLLaI Mal-Eng (Pondicherry University, CC BY 4.0, DOI 10.17632/tzrcrrwz4n.1). COMI-LINGUA, LinCE, IndicTriMix, Bhasha-Abhijnaanam (CC0) and DravidianCodeMix validation split (CC-BY 4.0) were also used for evaluation only.

License

Apache-2.0. Copyright 2026 Arun Prabhakaran. Fine-tuned from FacebookAI/xlm-roberta-base (MIT); see LICENSE and NOTICE.

More from the author

Built by Arun Prabhakaran (Latent AI Labs). Open models for Indian languages:

• Yama: zero-shot text classifier
• Chitragupta: PII detection and redaction
• Nandi: complaint routing

Write-up: My Model Thought Tamil Was Telugu · Summary and FAQ: https://latentailabs.github.io/bhasha-language-id/

• Articles and FAQs: latentailabs.github.io
• Medium: @arunp0
• YouTube: @LatentAILabs — how AI works, measured on real GPUs
• GitHub: arunp0 · LinkedIn: arunp0