Sakura — Jais-3-7B (GGUF)
GGUF quantizations of Jais-3-7B (Arabic and English, dense, K2 Horizon architecture, 36 layers, 4096 hidden size, context up to 524,288 tokens by its config). The official GGUF repository IFM/Jais-3-7B-GGUF already exists; this repository differs in that every file up to 5 bits was made with an importance matrix (calibration text below) and in what we measured. The version with edited weights is Sakura-Jais-3-7B-Uncensored-GGUF.
What was done
- We converted the original
safetensorscheckpoint (inception42/Jais-3-7B, revision94d69f49) to a BF16 GGUF withconvert_hf_to_gguf.pyof a llama.cpp checkout of 2026-10-07 (commit9c2e0e491), then quantized withllama-quantizeof llama.cpp b11594. - The importance matrix was computed on the BF16 file with
llama-imatrix(96 chunks of 512 tokens of ourcalib_mix_v1text mix, which contains English and German but no Arabic). Q8_0 and Q6_K were made without it. - No weights were edited.
Files
| File | Size | What it is |
|---|---|---|
Sakura-Jais-3-7B-Q8_0-8.92GiB.gguf |
8.92 GiB | Q8_0 (no importance matrix) |
Sakura-Jais-3-7B-Q6_K-6.89GiB.gguf |
6.89 GiB | Q6_K (no importance matrix) |
Sakura-Jais-3-7B-Q5_K_M-6.02GiB.gguf |
6.02 GiB | Q5_K_M with importance matrix |
Sakura-Jais-3-7B-Q4_K_M-5.21GiB.gguf |
5.21 GiB | Q4_K_M with importance matrix |
Sakura-Jais-3-7B-IQ4_XS-4.76GiB.gguf |
4.76 GiB | IQ4_XS with importance matrix |
Run
llama-server -m Sakura-Jais-3-7B-Q5_K_M-6.02GiB.gguf --jinja -ngl 99 -c 8192
Jais-3 is a model that always thinks: its chat template opens <ifm|think>, <ifm|think_fast> or <ifm|think_faster> for reasoning_effort high (default), medium or low. Pass the effort per request as chat_template_kwargs: {"reasoning_effort": "low"}. The architecture (K2 Horizon) needs a llama.cpp build with the support merged on 2026-10-06 (PR 29535); we used b11594, older builds do not know it. enable_thinking is not used by this template.
What we measured
Distance to Q8_0 (KL divergence of the next-token distribution of each file against our Q8_0 of the same model, llama-perplexity b11594, 20 chunks of 2048 tokens of an English (en) and a web (web) evaluation text; mean ± standard error; the reference is Q8_0, not the BF16 file). The self-check (Q8_0 against itself) gave 99.995 % same top token and KLD 0.
| File | Size | KLD en | KLD web | same top token en / web |
|---|---|---|---|---|
| Q6_K | 6.89 GiB | 0.0157 ± 0.0006 | 0.0072 ± 0.0002 | 94.9 / 95.3 % |
| Q5_K_M | 6.02 GiB | 0.0138 ± 0.0008 | 0.0073 ± 0.0001 | 95.1 / 95.1 % |
| Q4_K_M | 5.21 GiB | 0.0320 ± 0.0012 | 0.0168 ± 0.0002 | 92.6 / 92.6 % |
| IQ4_XS | 4.76 GiB | 0.0601 ± 0.0022 | 0.0234 ± 0.0003 | 90.6 / 91.4 % |
The perplexity of the Q8_0 file on the same texts is 8.51 ± 0.18 (en) and 12.69 ± 0.24 (web). Q5_K_M (with importance matrix) is not further from Q8_0 than Q6_K (without); the 4-bit files are roughly two to three times (Q4_K_M) and three to four times (IQ4_XS) as far from Q8_0 as Q5_K_M, so we would take Q5_K_M or larger when memory allows.
Smoke test (Q8_0 and the uncensored Q8_0 of the sister repository, llama-server b11594, reasoning_effort low, temperature 0): three short questions, one English and two Arabic; all answers were coherent and in the language of the question.
Not measured: anything in Arabic (quality of the quantizations, perplexity, benchmarks), any capability benchmark, speed, long contexts, tool calls, the high and medium reasoning efforts.
License
Apache-2.0 as the original model (LICENSE). See NOTICE.
Credits
- Model: inception42/Jais-3-7B and the official GGUF IFM/Jais-3-7B-GGUF.
- Quantization and measurement: ggml-org/llama.cpp (MIT).
中文版见 README_zh.md。