Dhīr 5B-A1.7B · 4-bit
Dhīr 5B-A1.7B quantised to 4-bit for llama.cpp, for English and Indian languages. At 3.4 GB, it runs on laptops, desktops, mobiles and embedded chips. Because only 1.7B of its parameters work on each token, it generates text faster than Dhīr 2B on a CPU.
- Original model: zenteiq-ai/Dhir-5B-A1.7B
- Parameters: 5.1B, of which 1.7B are active for each token
- Context length: 16,384 tokens
- File:
Dhir-5B-A1.7B-Q4_K_M.gguf(3.4 GB) - License: Apache 2.0
Choose a version
| Version | Size | Best for |
|---|---|---|
| zenteiq-ai/Dhir-2B-4bit | 1.3 GB | Edge devices, mobiles and embedded chips with little memory |
| zenteiq-ai/Dhir-2B-8bit | 2.2 GB | Laptops and mobiles, closest to the original |
| Prathat/Dhir-5B-A1.7B-4bit | 3.4 GB | Laptops and desktops; stronger and faster than the 2B on a CPU |
| zenteiq-ai/Dhir-5B-A1.7B-8bit | 5.4 GB | Laptops, desktops and small GPUs, closest to the original |
Quickstart
Until Dhīr support is merged into llama.cpp, build the dhir-support branch:
git clone -b dhir-support https://github.com/zenteiq-research/llama.cpp
cd llama.cpp
cmake -B build # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build -j --target llama-server llama-cli
Then run the model by its repository name; llama.cpp downloads the file on first use:
# Server with an OpenAI-compatible API on port 8080
./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080
# Chat in the terminal
./build/bin/llama-cli -hf Prathat/Dhir-5B-A1.7B-4bit --jinja
On a CPU, set -t to your number of performance cores and -tb to your total number of cores for the best speed, for example -t 8 -tb 24.
Python (LangChain)
pip install -U langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="http://localhost:8080/v1",
api_key="EMPTY",
model="Prathat/Dhir-5B-A1.7B-4bit",
temperature=0.27,
top_p=0.98,
presence_penalty=0.25,
frequency_penalty=0,
extra_body={"min_p": 0.06, "repeat_penalty": 1.025},
)
print(llm.invoke("भारत की राजधानी के बारे में दो वाक्य लिखिए।").content)
Memory
The model needs about 3.4 GB plus 0.07 MB for each token of conversation. For long conversations on devices with little memory, an 8-bit context cache halves the second part:
./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080 -fa on -ctk q8_0 -ctv q8_0
Quality
We tested this file against the original model on WikiText-2 perplexity, on the KL divergence of its next-token predictions over 1.5 million tokens in 11 languages, code and mathematics, and on standard benchmarks. Its perplexity is within 1.8% of the original, and it picks the same next token 91% of the time.
For the full evaluation, see our technical paper and zenteiq.ai/research/blog.
Limitations
- 4-bit quantisation gives up a little accuracy for its size; use the 8-bit version when memory allows.
- For multi-step arithmetic in Bengali and Telugu, prefer the 8-bit version.
- Like any language model, Dhīr can state incorrect facts fluently. Verify what matters, and add your own safeguards in production.
Files
| File | Size | SHA-256 |
|---|---|---|
Dhir-5B-A1.7B-Q4_K_M.gguf |
3.4 GB | 6e137eb3d1713419c73c9a4cad1ea6e851f6f9aa9c361c4715492c62cda5e1c9 |
License
Apache 2.0, the same as the original model. The LICENSE and NOTICE files of zenteiq-ai/Dhir-5B-A1.7B are included in this repository.