A sparse Mixture-of-Experts foundation model, designed and pretrained from scratch in India, with an in-house tokenizer built for Indian languages.
ZenteiQ · AIKosh · BrahmAI family · Apache 2.0
Dhīr 5B MoE is the higher-capacity model in ZenteiQ's BrahmAI family. It was designed and pretrained from scratch by ZenteiQ's team in India, rather than adapted from an existing foundation model. It understands English and ten Indian languages. Its sparse Mixture-of-Experts design holds 5.09B parameters, but each token activates only 1.79B of them: two of twelve experts in every layer. This gives it more capacity than a dense model of the same per-token cost, for enterprise workloads on a single GPU or inside a private infrastructure boundary.
| Developer | ZenteiQ AiTech Innovations Pvt. Ltd., Bengaluru, India |
| Model type | Sparse Mixture-of-Experts decoder-only Transformer, instruction-tuned (SFT and GRPO reinforcement learning) |
| Parameters | 5.09B total, 1.79B active per token |
| Experts | 12 routed experts per layer, top-2 routing, no shared experts |
| Languages | English, Hindi, Gujarati, Tamil, Bengali, Kannada, Malayalam, Marathi, Odia, Punjabi, Telugu |
| Context length | 16,384 tokens |
| Licence | Apache 2.0 |
| Support | IndiaAI Mission, MeitY, under the Foundation Model initiative |
| Companion model | Dhīr 2B (dense, for lightweight and on-device deployment) |
Dhīr 2B is optimised for Indic document QA and lightweight, on-device deployment; Dhīr 5B MoE adds stronger reasoning, code and tool calling at ~1.8B active-parameter cost.
| Component | Configuration |
|---|---|
| Parameters | 5.09B total, 1.79B active per token |
| Layers | 28 |
| Hidden dimension | 2560 |
| Attention | Grouped-Query Attention: 20 query heads, 5 key/value heads, head dim 128, QK normalisation |
| Feed-forward | Mixture of Experts: 12 routed experts, top-2 routing, expert intermediate dimension 1536 |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| Normalisation | RMSNorm |
| Attention kernel | Flash Attention |
| Vocabulary | 131,072 (in-house tokenizer for Indian languages) |
| Context | 16k tokens |
| Precision | bfloat16 |
Training stack: MaxText (JAX) on Google TPU v6e. Checkpoints are trained in Orbax format and converted to Hugging Face format for release.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "zenteiq-ai/Dhir-5B-A1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
messages = [{"role": "user", "content": "भारत की राजधानी के बारे में दो वाक्य लिखिए।"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
output = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
trust_remote_code=True is required. Dhīr ships its architecture as custom code, included in this repository, so Transformers must be allowed to run it when loading the model. The same applies to other tools that load the model through Transformers: for example, pass --trust-remote-code to vllm serve.
| Use | Notes |
|---|---|
| Indic ↔ English translation | Drafting and everyday communication. Human review is recommended for published or legal content. |
| Question answering over supplied documents (RAG reader) | Answers from a passage you provide, in English and Indian languages. |
| Indian-language text understanding | Sentiment, reading comprehension and summarisation. |
| Base model for Indic fine-tuning | A strong starting point for organisations building their own Indian-language model with MoE efficiency. |
| Basic Python assistance | Short, self-contained functions. |
| Tool and function calling | Native function calling for assistants that call tools and APIs. |
| Private and on-premises AI | Enterprise assistants and privacy-sensitive applications on local infrastructure. |
All figures are for the Dhīr 5B MoE release checkpoint, from ZenteiQ's internal benchmark suite (2 November 2026). Ranks are against open models of up to 3.2B parameters, comparable to the 1.79B parameters Dhīr 5B MoE activates per token, run on the same benchmark and harness: 11 instruction-tuned models for generative tasks and 12 models (base and instruct) for log-likelihood tasks, Dhīr 5B MoE included. Ranks exclude the other Dhīr model. Leads of less than 1 point are reported as joint first: differences that small are within run-to-run and evaluation-framework variation.
| Capability | Benchmark | Dhīr 5B MoE | Next best | Rank |
|---|---|---|---|---|
| Translation, 24 Indic ↔ English directions (chrF) | IndicGenBench FLORES | 50.0 | 33.4 | #1 of 11 |
| Cross-lingual summarisation, 13 languages | IndicGenBench CrossSum | 18.5 | 15.2 | #1 of 11 |
| Sentiment, 14 languages (accuracy) | IndicSentiment | 96.1 | 88.7 | #1 of 12 |
| Causal reasoning, 19 languages (accuracy) | IndicCOPA | 62.8 | 60.3 | #1 of 12 |
| Document QA, 13 languages (token F1) | IndicGenBench XQuAD-IN | 45.3 | 41.8 | #1 of 11 |
| Cross-lingual QA, 13 languages (token F1) | IndicGenBench XorQA-IN | 26.7 | 20.5 | #1 of 11 |
| Reading comprehension, 20 languages (accuracy) | Belebele | 60.1 | 49.5 | #1 of 12 |
| Story completion (accuracy) | XStoryCloze | 68.3 | 67.0 | #1 of 12 |
| Indian knowledge (accuracy) | INCLUDE / MILU | 40.0 / 41.6 | 37.6 / 41.5 | #1 of 12 / Joint #1 of 12 |
| Instruction following, English (strict) | IFEval | 70.6 | 70.4 | Joint #1 of 11 |
| Python (pass@1) | MBPP + HumanEval | 56.5 | 44.4 | #1 of 12 |
| Grade-school maths, English | GSM8K | 70.8 | Basic maths reasoning | |
| Grade-school maths, Indian languages | MGSM | 45.6 | ||
| Function calling, native tool format (accuracy) | BFCL v3, AST checker | 76.3 |
Reference: larger models. Dhīr 5B MoE also scores above every 4B-class model in the comparison (Gemma-3-4B-it, Qwen3-4B, Qwen3-4B-2507, Phi-4-mini, Phi-3-mini, Phi-3.5-mini) on translation, cross-lingual summarisation and sentiment, and above Gemma-3-4B-it on code (43.4).
Document QA is measured when the answer is in a supplied passage. Function calling is scored in the model's native tool format (BFCL v3, AST checker), averaged over 15 categories.
Deployment. Weights take 10.19 GB in BF16. Quantised variants, hardware validation and serving throughput will be published with the Dhīr Technical Report.
Evaluation notes
Dhīr 5B MoE is a compact model, and we recommend the following for responsible deployment:
Dhīr 5B MoE has been evaluated for safety behaviour, including probing with the NVIDIA Garak vulnerability scanner across jailbreak, prompt-injection and harmful-request categories. Detailed results will be published with the technical report. Deployers should add application-level safeguards suited to their use case.
Data composition, training compute and methodology will be provided in the Dhīr Technical Report.
Released under the Apache 2.0 licence through AIKosh, Hugging Face and ZenteiQ.
ZenteiQ acknowledges the support of the IndiaAI Mission, Ministry of Electronics and Information Technology (MeitY), under the Foundation Model initiative.
@misc{zenteiq2026dhir,
title = {Dh\={\i}r : Sovereign Compact Foundation Models },
author = {{ZenteiQ AiTech Innovations Pvt. Ltd.}},
year = {2026},
howpublished = {Technical report forthcoming}
}
ZenteiQ | India
model-00002-of-00004.safetensors 3.2 GBmodel-00003-of-00004.safetensors 3.2 GBmodel-00001-of-00004.safetensors 3.2 GBmodel-00004-of-00004.safetensors 545 MB