Mitra IT Qwen3.5-9B — GGUF quantizations (unofficial)

These are unofficial requantizations. The original model, its model card, the evaluation numbers and the training recipe are published by buddhist-nlp at https://huggingface.co/buddhist-nlp/mitra-qwen35-it — please read that card, it is the authoritative documentation. This repo only provides this model in quantized GGUF form so that it can be run with llama.cpp tooling (the llama-cli / llama-server binaries, llama-swap, LM Studio, KoboldCpp, …) or ollama on a typical GPU or on CPU.

Everything in this repo is derived from, and fully credited to, MITRA-QWEN / the Dharmamitra project.

---

The model

MITRA-QWEN-IT (mitra-qwen35-it) is a 9B model for Buddhist studies conversations, built on Qwen/Qwen3.5-9B. # It is the translation general-purpose instruction/chat model of the Dharmamitra project. It can be used for closed-book Buddhism Q&A plus translation and translation-refinement assistance for classical languages (Sanskrit, Tibetan, Buddhist Chinese, Pāli).

This is the recommended entry point if you want to talk to the mitra family.

If you are looking for a variant of this model that is meant for translation only, then look for the mitra-qwen35-translate variant of this model.

Architecture (unchanged from the base model, general.architecture = qwen35): 32 layers, hidden size 4096, intermediate 12288, 16 Q / 4 KV heads, head dim 256, hybrid linear + full attention (full_attention_interval = 4), partial RoPE (θ = 1e7, factor 0.25), vocab 248 320, context 262 144 tokens, tie_word_embeddings = false.

---

Files in this repo

---

Which quant should I pick?

• Q8_0 if you have ≥ 12 GB free VRAM and enough processing time, use this one.
• Q6_K if you have 8GB free VRAM.
• Q5_K_M if Q6_K is too big or still too slow.
• Q4_K_M if you are low on VRAM or want the fastest response times.

KV cache is a large part of the budget at long context: use --cache-type-k q8_0 --cache-type-v q8_0 or use a lower --ctx-size if you do not need the full 262 k window.

---

How to prompt it

What is the difference between śamatha and vipaśyanā?

---

llama-cli example

The sampling parameters below are the settings that work well for this model (temperature 0.2, top-k 40, top-p 0.1, min-p 0.05, repeat-penalty 1.05 — the repetition_penalty 1.05 recommended in the original card suppresses the rare long-transliteration repetition loops; the very small top-p keeps the greedy-ish, deterministic behaviour that the evaluation was run with).

MODEL=Mitra-Translate-Qwen3.5-9.0B-Q5_K_M.gguf
PROMPT=$(cat <<'TXT'
What is the difference between śamatha and vipaśyanā?
TXT
)

llama-cli \
  --model "$MODEL" \
  --reasoning off \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --ctx-size 32768 \
  -b 2048 -ub 2048 \
  -fa on \
  --temp 0.2 --top-k 40 --top-p 0.1 --min-p 0.05 --repeat-penalty 1.05 \
  -p "$PROMPT"

Notes:

• --reasoning off plus --chat-template-kwargs '{"enable_thinking": false}' are both required — the first keeps llama.cpp from expecting a thinking block, the second selects the non-thinking branch of the embedded Qwen3.5 chat template. This model does not support thinking and will not work correctly unless you disable the thinking mode with these two arguments!
• Do not add --spec-type draft-mtp (no MTP tensors in these files, so MTP mode is not supported). If you do want to use MTP with this model, you can use the separate MTP model head from christian-steinert/mtp-Qwen3.5-9B-GGUF.

For a longer conversations raise --ctx-size (the model supports 262 144) and, if VRAM is tight, --fit on --fit-ctx 8192 or --cache-type-k q8_0 --cache-type-v q8_0.

---

Credit, acknowledgement and license

• Original model: buddhist-nlp/mitra-qwen35-translate — MITRA-QWEN / Dharmamitra project. Its model card, evaluation tables and training description are the primary source; this README only summarises them.
• Base model: Qwen/Qwen3.5-9B.
• License: Apache-2.0, inherited from the original release. These GGUF files are requantized versions of the same weights; no change of license is intended or implied.
• The original authors ask (as a kind request, not a requirement) for a short credit to the Dharmamitra / MITRA project with a link to https://dharmamitra.org when the model supports your work — including when you use one of these quantizations. Please do by citing the Dharmamitra / MITRA project if you use the model: https://dharmamitra.org.

Known limitations

• This model requires thinking disabled as described aove.
• Additionally, in these GGUF files: no MTP speculative-draft head (see note above), and quantization adds a small amount of noise on top of the bf16 originals — use Q8_0 if you aim for nearly reference quality.