Muslim-6B-PRO — GGUF

GGUF quantizations of NightPrince/Muslim-6B-PRO for local inference with llama.cpp and compatible runtimes (LM Studio, Ollama, koboldcpp, etc.).

Files

Usage

llama.cpp

llama-cli -hf NightPrince/Muslim-6B-PRO-GGUF:Q4_K_M -p "السلام عليكم"

or, with a locally downloaded file:

llama-cli -m Muslim-6B-PRO-Q4_K_M.gguf -p "السلام عليكم"

Server mode (OpenAI-compatible API)

llama-server -m Muslim-6B-PRO-Q4_K_M.gguf --port 8080

Tool calling

This model uses the Hermes-style <tool_call> format. llama-server supports this natively via its --jinja flag (uses the model's built-in chat template) combined with the standard OpenAI-style tools parameter in requests.

Choosing a quant

• Q4_K_M is the recommended default for most use — the standard "good enough for almost everyone" tradeoff.
• Go Q5_K_M or Q6_K if you have the VRAM/RAM headroom and want output closer to the original fp16 model, especially for tool-call argument precision.
• Go Q2_K/Q3_K only under tight memory constraints — expect noticeably more degradation on precise tasks like tool-call JSON formatting and exact surah/ayah references.
• Q8_0 or F16 for maximum fidelity when memory isn't a constraint.

Related resources

• Full-precision model card, training details, and dataset: NightPrince/Muslim-6B-PRO
• Live demo with real tool-calling (ZeroGPU): NightPrince/muslim-6b-pro-demo
• Fine-tuning code: github.com/NightPrinceY/Karnak-6B-Finetuning