Dhīr 5B-A1.7B · GGUF
Dhīr 5B-A1.7B quantised for llama.cpp, for English and Indian languages. This repository holds a 4-bit and an 8-bit build. At 3.4 GB, the 4-bit build runs on laptops, desktops, mobiles and embedded chips; because only 1.7B of its parameters work on each token, it generates text faster than Dhīr 2B on a CPU. At 5.4 GB, the 8-bit build runs on laptops, desktops and small GPUs, with output almost identical to the original model.
ZenteiQ · AIKosh · BrahmAI family · Apache 2.0
- Original model: zenteiq-ai/Dhir-5B-A1.7B
- Parameters: 5.1B, of which 1.7B are active for each token
- Context length: 16,384 tokens
- Files:
Dhir-5B-A1.7B-Q4_K_M.gguf(3.4 GB),Dhir-5B-A1.7B-Q8_0.gguf(5.4 GB) - License: Apache 2.0
Choose a version
| Version | Size | Best for |
|---|---|---|
zenteiq-ai/Dhir-2B-GGUF:Q4_K_M |
1.3 GB | Edge devices, mobiles and embedded chips with little memory |
zenteiq-ai/Dhir-2B-GGUF:Q8_0 |
2.2 GB | Laptops and mobiles, closest to the original |
zenteiq-ai/Dhir-5B-A1.7B-GGUF:Q4_K_M |
3.4 GB | Laptops and desktops; stronger and faster than the 2B on a CPU |
zenteiq-ai/Dhir-5B-A1.7B-GGUF:Q8_0 |
5.4 GB | Laptops, desktops and small GPUs, closest to the original |
Quickstart
Until Dhīr support is merged into llama.cpp, build the dhir-support branch:
git clone -b dhir-support https://github.com/zenteiq-research/llama.cpp
cd llama.cpp
cmake -B build # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build -j --target llama-server llama-cli
Then run the model by its repository name and build; llama.cpp downloads the file on first use:
# Server with an OpenAI-compatible API on port 8080
./build/bin/llama-server -hf zenteiq-ai/Dhir-5B-A1.7B-GGUF:Q4_K_M --jinja --port 8080
# Chat in the terminal
./build/bin/llama-cli -hf zenteiq-ai/Dhir-5B-A1.7B-GGUF:Q4_K_M --jinja
Use :Q8_0 in place of :Q4_K_M for the 8-bit build.
On a CPU, set -t to your number of performance cores and -tb to your total number of cores for the best speed, for example -t 8 -tb 24.
Python (LangChain)
pip install -U langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="http://localhost:8080/v1",
api_key="EMPTY",
model="zenteiq-ai/Dhir-5B-A1.7B-GGUF",
temperature=0.27,
top_p=0.98,
presence_penalty=0.25,
frequency_penalty=0,
extra_body={"min_p": 0.06, "repeat_penalty": 1.025},
)
print(llm.invoke("भारत की राजधानी के बारे में दो वाक्य लिखिए।").content)
Memory
The model needs about 3.4 GB (4-bit) or 5.4 GB (8-bit) plus 0.07 MB for each token of conversation. For long conversations on devices with little memory, an 8-bit context cache halves the second part:
./build/bin/llama-server -hf zenteiq-ai/Dhir-5B-A1.7B-GGUF:Q4_K_M --jinja --port 8080 -fa on -ctk q8_0 -ctv q8_0
Quality
We tested each file against the original model on WikiText-2 perplexity, on the KL divergence of its next-token predictions over 1.5 million tokens in 11 languages, code and mathematics, and on standard benchmarks.
| Build | Perplexity | Same next token as the original |
|---|---|---|
4-bit (Q4_K_M) |
within 1.8% of the original | 91% of the time |
8-bit (Q8_0) |
within 0.2% of the original | 98% of the time |
For the full evaluation, see our technical paper and zenteiq.ai/research/blog.
Limitations
- 4-bit quantisation gives up a little accuracy for its size; use the 8-bit build when memory allows.
- For multi-step arithmetic in Bengali and Telugu, prefer the 8-bit build.
- Like any language model, Dhīr can state incorrect facts fluently. Verify what matters, and add your own safeguards in production.
Files
| File | Size | SHA-256 |
|---|---|---|
Dhir-5B-A1.7B-Q4_K_M.gguf |
3.4 GB | 6e137eb3d1713419c73c9a4cad1ea6e851f6f9aa9c361c4715492c62cda5e1c9 |
Dhir-5B-A1.7B-Q8_0.gguf |
5.4 GB | 864d24e8572c5d781272d740ca9f6b4e56343f496dc59442a44508ebcb8d1945 |
License
Apache 2.0, the same as the original model. The LICENSE and NOTICE files of zenteiq-ai/Dhir-5B-A1.7B are included in this repository.
Availability
Released under the Apache 2.0 licence through AIKosh, Hugging Face and ZenteiQ.
Acknowledgement
ZenteiQ acknowledges the support of the IndiaAI Mission, Ministry of Electronics and Information Technology (MeitY), under the Foundation Model initiative.
Citation
@misc{zenteiq2026dhir,
title = {Dh\={\i}r : Sovereign Compact Foundation Models },
author = {{ZenteiQ AiTech Innovations Pvt. Ltd.}},
year = {2026},
howpublished = {Technical report forthcoming}
}
Contact
ZenteiQ | India