laya-indic

A newer version is available: Yama (v3, XLM-RoBERTa-large encoder), released as a separate model. It scores 0.862 against 0.800 for this model on the same 67 tasks and is better calibrated. This repository stays at v2 (mmBERT-base); v1 is at revision v1.0.

A multilingual typed-decision model with a focus on Indian languages and Hinglish. You give it a text (a message, ticket, review, headline or document) and one or more questions written in plain English. It answers every question in a single forward pass, with a calibrated probability for each option. Question schemas are defined at inference time, so it handles new labels and new tasks without retraining.

laya-indic is an independent personal release. It is not affiliated with or endorsed by HAL-X AI. It uses the THX-01 decision architecture and the open-source thx01 library (Apache-2.0, HAL-X AI), but it is built on the jhu-clsp/mmBERT-base encoder (MIT), not on THX-01's weights. The encoder's pretraining is first continued on public Indian-language web text (Sangraha, CC-BY-4.0), and then the decision model is trained. All training data is public and permissively licensed, plus labels generated by open-weights Qwen3 models.

What's new in v2

• Continued pretraining on Indian languages. Before decision training, the mmBERT-base encoder ran one masked-language-modelling pass over 457M tokens of Sangraha (11 Indian languages plus English) and Hinglish replay. Held-out MLM loss dropped 7–25% per language.
• Real Indian-language text. 23,955 snippets of real Sangraha web text in 11 languages, with 63,448 questions about them written and verified by open-weights Qwen3. v1 saw only synthetic Indic text.
• Results. Overall 0.789 → 0.800. SIB-200 Indic topic 0.626 → 0.670, up in 9 of 10 languages. SIB-200 in THX-01's languages 0.751 → 0.779. Other families are flat, except IndicSentiment, which dipped (0.892 → 0.884). The earlier weights are kept at revision v1.0.

Question types

THX-01 also supports number, excerpt and "cite": true. This model was not trained for those and does not support them.

Quickstart

import thx01   # pip install thx01

agent = thx01.load("arunp0/laya-indic")          # or a local directory; GPU if available
# v1 weights: thx01.load(huggingface_hub.snapshot_download("arunp0/laya-indic", revision="v1.0"))

text = "Kal se mera UPI payment fail ho raha hai, paise kat gaye par merchant ko nahi mile"
out = agent.system_one(text, {
    "topic":  {"type": "choice", "instructions": "What is the customer writing about?",
               "criteria": {"failed_payment": "payment failed or money debited but not received",
                            "refund": "asking for a refund", "account": "login or account access", "other": None}},
    "urgent": {"type": "noul", "instructions": "Does the customer need help urgently?"},
    "anger":  {"type": "score", "instructions": "How frustrated is the customer?",
               "criteria": ["calm", "mildly annoyed", "clearly frustrated", "furious"]},
})
# topic  -> failed_payment (0.97)
# urgent -> P(yes) 0.55
# anger  -> "clearly frustrated" 0.85

The response's model field says "THX-01". That is the thx01 library's default label; the weights are this model's.

For many texts, use agent.predict_batch(texts, questions, batch_size=64). questions can be one schema shared by every text or a list with one schema per text.

Write questions in English and leave the text in its own language and script. That is how the model was trained.

Benchmarks

Zero-shot accuracy on 67 tasks, compared with THX-01 (322M, same architecture, roughly 2.1M training decisions) and with laya-indic v1. Both models were scored with the same harness, the same prompts and the same test samples. The harness is adapted from THX-01's open benchmark code (bench/tasks.py).

Per task: v2 is ahead of THX-01 on 47 of 67 tasks, behind on 19, tied on 1; ahead of v1 on 40, behind on 24, tied on 3.

Read these numbers with the training data in mind. Neither model saw these test sets. Some tasks, however, have their training splits in one model's data and not in the other's:

All training texts, including the continued-pretraining corpus (line by line), were deduplicated against every evaluation set, using normalised exact match.

Per-task results (accuracy and expected calibration error)

ECE is expected calibration error with 15 equal-width bins; lower is better. Mean ECE over the 67 tasks is 0.091 for v2, 0.087 for v1 and 0.098 for THX-01.

‡ BoolQ-hi is left out of every mean. Our harness had a label-parsing bug in the first runs of this benchmark. The row shows THX-01 rescored after the fix (0.643, n = 1000) next to v1 and v2, which were scored after the fix.

Where v2 is stronger than THX-01:

• Indian-language intent and domain (MASSIVE Indic +0.18).
• Hinglish (+0.15).
• Banking77 (+0.16), SMS spam and emotion.
• SIB-200 topic in Indian languages (+0.04) and Kazakh.
• New decision schemas it has never seen (synthetic held-out +0.16).

Where it is weaker:

• AG News (0.864 vs 0.931). THX-01 reports training on public benchmarks' training splits; AG News is excluded here.
• SIB-200 in English and German.
• IndicCOPA, a causal-reasoning task, where both models are close to chance.

v2 vs v1: v2 is ahead on 40 tasks and behind on 24. The gains are concentrated in SIB-200 topic classification (Indic +0.044, others +0.028). The largest drop is Tamil IndicSentiment (0.900 → 0.847).

Training

Method. RLCD (Reinforcement Learning for Calibrated Decisions), as introduced by THX-01 and implemented in thx01:

• Policy-gradient term: sampling from noisy logits, rewarded by a strictly proper scoring rule.
• Soft cross-entropy term: on label-smoothed targets.
• Calibration: a temperature for each question type, fitted after training on held-out data.

Stage 1: continued pretraining (new in v2). jhu-clsp/mmBERT-base ran one pass of masked-language modelling (15% masking, sequence length 512, learning rate 3e-5) over 457M tokens:

• Sangraha "verified" documents in Telugu, Gujarati, Kannada, Malayalam, Punjabi, Hindi, Bengali, Tamil, Marathi, Urdu, Odia and English.
• A small replay of Hinglish texts from the training data below.

Every line that matched an evaluation text was removed. Stage 2: decision training from that encoder, with a freshly initialised decision head.

Public data (285,078 decisions; every source is CC0, CC-BY, Apache-2.0 or MIT; full attribution in LICENSES.md):

Synthetic data (173,226 verified questions over 62,263 texts). The generator is Qwen3-235B-A22B-Instruct-2507 and the verifier is Qwen3-30B-A3B-Instruct-2507, both Apache-2.0. The pipeline is label-first:

1. The generator proposes a decision schema.
2. Gold answers are sampled.
3. The generator writes a text in a chosen language, script and register that makes those answers true.
4. The verifier answers blind, and a question is kept only if the verifier agrees with the gold answer (about 72% were kept).

About 68% of the texts are in Indian languages or Hinglish. As far as can be verified, no proprietary-model output was used. The one exception that can't be fully checked is NVIDIA's Nemotron Safety Guard v3, which is partly machine-translated and synthetic and doesn't name the models used; see LICENSES.md. The dataset is published as arunp0/laya-indic-synthetic; see also SYNTHETIC_DATA.md.

Real Indian-language text (new in v2). 23,955 snippets of 1–3 sentences or a short paragraph were cut from Sangraha documents (AI4Bharat, CC-BY-4.0):

For each snippet:

1. Qwen3-235B picked the topic from a fixed 16-topic list. It also wrote 2–3 further questions specific to the snippet (choice, yes/no or score).
2. The topic options were assembled in code. In 7% of cases the correct topic was removed, so "none of the options" is the answer.
3. Qwen3-30B answered every question blind, and only questions where the two models agreed were kept: 63,448 in training.

The data is published as arunp0/laya-indic-real-text (CC-BY-4.0).

Limitations

• Text classification only. It does not do number, excerpt or citation questions, open-ended generation or multi-step reasoning. IndicCOPA results show it cannot do commonsense causal reasoning.
• Instructions in English. Questions and option descriptions should be in English; the text can be in any supported language.
• Long texts are truncated at 1,024 tokens per question.
• Calibration is good on average but not everywhere. Recalibrate on your own data before using probabilities as thresholds.
• About 45% of the training labels come from Qwen3. 33% are questions on synthetic texts and 12% are questions on real web text. Synthetic Indic text is cleaner and more formal than real user text. Sangraha is mostly news and web prose, not chat. Urdu and Odia have little real text.
• Content warning. The safety and toxicity datasets (Aegis 2.0, Nemotron Safety Guard, Measuring Hate Speech, Civil Comments, Aya Red-teaming) contain offensive and harmful text. The model can classify such content but is not a tested content-moderation system.

Acknowledgements and licence

• Architecture, training objective and inference library: THX-01 / thx01 by HAL-X AI (Apache-2.0).
• Benchmark harness: THX-01's bench/tasks.py, including its MASSIVE scenario descriptions and label humaniser, which were also used in some training instructions.
• Encoder: mmBERT-base, by JHU CLSP (MIT).
• Indian-language corpus: Sangraha, by AI4Bharat (CC-BY-4.0), used for continued pretraining and real-text decisions.

The model is released under the Apache License 2.0. See NOTICE for the required attributions and LICENSES.md for every dataset's authors and licence.