A family of nine GGUF files of Qwen3.8-27B (the native vision-language 27B dense model, Gated DeltaNet + Gated Attention hybrid layout, 262,144-token native context, MTP speculative head), converted from unsloth/Qwen3.8-27B-NVFP4. The MTP (multi-token prediction) speculative head is baked into every file — no separate drafter needed.
ORIG — the source-preserving conversion: native NVFP4 MLP backbone + BF16 attention/embeddings (the source's F8 attention is dequantized to BF16 because GGML has no F8 tensor type). This is the largest file and the one all tiers are derived from.VERY-LOW / COMPACT-LOW / LOW / MEDIUM / MID-HIGH / HIGH / VERY-HIGH — a compact family sharing a byte-identical 448-tensor NVFP4 backbone (all attention + MLP re-quantized to NVFP4), differing only in the 10 "extra" tensors (LM head, token embedding, MTP draft head). COMPACT-LOW fills the gap between VERY-LOW and LOW — slightly smaller than LOW while keeping a materially stronger LM head than VERY-LOW (Q4_K vs Q3_K). MID-HIGH sits between MEDIUM and HIGH with all three head groups at Q8_0.HIGHEST — the top tier: keeps the source's native NVFP4 MLP (layers 0-55) exactly as in ORIG, restores Q8_0 for attention + the late MLP layers + the LM head, and keeps the token embedding + MTP head in BF16. The closest compact approximation of the source layout, for high-end GPUs.The goal: keep native NVFP4 density across the whole model for Blackwell, and offer a size/precision ladder for the tensors that most affect output quality and decode speed. On our dual 16 GB Blackwell setup every tier fits and runs (see notes before treating any numbers as meaningful).
Vision works. The model is a native VLM (images and video). Pair any of these GGUFs with the mmproj-BF16.gguf included in this repo (byte-identical to unsloth/Qwen3.8-27B-GGUF's projector) via --mmproj.
For 16 GB VRAM users who want NVFP4 as small as possible: the same NVFP4 backbone, squeezed further — the MTP head stripped out and the LM head / embeddings trimmed to the absolute minimum — is published as
BUDGET/STARVEDin esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUF.
I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.
🌐 esatapedico.com. Updates, work-in-progress, and ways to support.
| File | Size | lm_head (output.weight) |
token_embd | MTP head (blk.64) | Attention |
|---|---|---|---|---|---|
Qwen3.8-27B-NVFP4-MTP-ORIG.gguf |
33.13 GB | BF16 |
BF16 |
BF16 |
BF16 (source F8 dequant) |
Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf |
14.86 GB | Q3_K |
Q2_K |
Q2_K |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf |
15.16 GB | Q4_K |
Q3_K |
Q2_K |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-LOW.gguf |
15.53 GB | Q5_0 |
IQ4_XS |
IQ4_XS |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf |
16.38 GB | Q8_0 |
Q6_K |
IQ4_XS |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf |
16.90 GB | Q8_0 |
Q8_0 |
Q8_0 |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-HIGH.gguf |
17.57 GB | BF16 |
Q6_K |
IQ4_XS |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf |
19.69 GB | BF16 |
BF16 |
BF16 |
NVFP4 |
Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf |
23.19 GB | Q8_0 |
BF16 |
BF16 |
Q8_0 |
mmproj-BF16.gguf |
931 MB | — (vision projector) |
ORIG (1,202 tensors): the direct --outtype auto conversion of the pre-processed source checkpoint.
| GGML type | Tensors | Size | Component |
|---|---|---|---|
NVFP4 |
168 | 8.42 GB | MLP gate/up/down of layers 0-55 (source NVFP4) |
BF16 |
338 | 24.69 GB | attention (self_attn + linear_attn, source F8 dequantized), MLP layers 56-63, LM head, embeddings, MTP head |
F32 |
696 | 0.01 GB | norms, scales |
VERY-LOW–VERY-HIGH (1,202 tensors): the 7 compact tiers share a byte-identical 448-tensor NVFP4 backbone (all 64 layers' attention + MLP, 13.69 GB) and differ only in the 10 extra tensors:
| GGML type | Tensors | Component |
|---|---|---|
NVFP4 |
448 | all 64 transformer blocks (attention QKV/output + FFN gate/up/down) — identical in all tiers |
F32 |
744 | norms, gates, scales |
| tier-dependent | 10 | output.weight, token_embd.weight, blk.64.* MTP draft block (incl. nextn.eh_proj) |
MID-HIGH (1,202 tensors): the same compact 448-tensor NVFP4 backbone with all three head groups at Q8_0 (LM head, token embedding, MTP draft head) — the highest-precision heads the compact backbone can carry. Same tensor layout as the table above, with the 10 tier-dependent tensors at Q8_0 (3.15 GB).
HIGHEST (1,202 tensors): mirrors the source layout — the NVFP4 MLP of layers 0-55 is byte-identical to ORIG, the BF16 tensors (attention, MLP 56-63, LM head) are restored to Q8_0, and the token embedding + MTP head stay BF16.
| GGML type | Tensors | Size | Component |
|---|---|---|---|
NVFP4 |
168 | 8.42 GB | MLP gate/up/down of layers 0-55 (byte-identical to ORIG) |
Q8_0 |
281 | 11.30 GB | attention (self_attn + linear_attn), MLP layers 56-63, LM head |
BF16 |
9 | 3.39 GB | token_embd, MTP head (blk.64) |
F32 |
744 | 0.01 GB | norms, gates, scales |
The MTP draft head is embedded in every GGUF (blk.64.nextn.*), so no separate drafter file is needed. Enable it in llama.cpp with --spec-type draft-mtp.
The source checkpoint, unsloth/Qwen3.8-27B-NVFP4, is a mixed-precision compressed-tensors model:
float-quantized, 8-bit): self_attn.q/k/v/o_proj, linear_attn.in_proj_qkv/in_proj_z/out_proj, lm_head, and the MLP of layers 56-63nvfp4-pack-quantized, 4-bit): the MLP of layers 0-55llama.cpp's converter does not yet accept a compressed-tensors checkpoint with more than one config group unless all are NVFP4 (it raises NotImplementedError). To convert this model we therefore:
weight × scale), removed their weight_scale tensors, kept the NVFP4-packed MLP tensors untouched, and pinned the config to a single nvfp4-pack-quantized group.convert_hf_to_gguf.py --outtype auto → ORIG (NVFP4 MLP preserved natively, F8 attention materialized as BF16).llama-quantize --tensor-type-file <overrides> from the ORIG parent: the 7 compact tiers re-quantize the BF16 attention to NVFP4 for a compact uniform backbone and pin the extra tensors per tier; MID-HIGH is the compact tier with all three head groups at Q8_0; HIGHEST instead restores attention/MLP56-63/lm_head to Q8_0 while keeping the NVFP4 MLP and BF16 embedding/MTP head.Honest trade-off note: the compact tiers re-quantize attention from BF16 (which was F8 in the source) down to NVFP4. That is a second quantization step on those tensors (F8 → BF16 → NVFP4) and does lose some attention precision vs the source. HIGHEST keeps the same NVFP4 MLP as ORIG and uses Q8_0 (instead of BF16) for the attention path — much closer to the source's effective precision at ~10 GB less than ORIG. The ORIG file preserves the source's attention quality at the cost of a 33 GB file. If you want the highest-fidelity version, use ORIG or HIGHEST; if you need the compact footprint, the tiers are ~2× smaller.
This is a derivative work built entirely from existing Apache-2.0 artifacts. Nothing here was trained or fine-tuned. Credit belongs to:
mmproj-BF16.gguf).The NVFP4 tensors are native GGML type 40, preserved from the source checkpoint through GGUF conversion (for the MLP backbone) with no re-quantization round trip.
unsloth/Qwen3.8-27B-NVFP4 (safetensors, mixed F8+NVFP4) was pre-processed by dequantizing F8→BF16 and keeping the NVFP4-packed tensors (see above) — this is the only step that touches values beyond type conversion, and only on the 233 F8 tensors.convert_hf_to_gguf.py --outtype auto → ORIG (33.13 GB).llama-quantize --tensor-type-file <overrides> from the ORIG parent. The per-tier override maps are in this repo (overrides-very-low.txt, overrides-compact-low.txt, overrides-low.txt, overrides-medium.txt, overrides-mid-high.txt, overrides-high.txt, overrides-very-high.txt, overrides-highest.txt, 1,202 entries each).The NVFP4 backbone is byte-identical across the family: the 448-tensor backbone of the seven compact tiers is the same set of bytes in every tier, and HIGHEST's 168 NVFP4 MLP tensors are byte-identical to ORIG's (verified: per-tensor SHA-256).
Qwen3.8-27B-NVFP4-MTP-ORIG.gguf (33.13 GB) — source-preserving conversionQwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf (14.86 GB)Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf (15.16 GB)Qwen3.8-27B-NVFP4-MTP-LOW.gguf (15.53 GB)Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf (16.38 GB)Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf (16.90 GB)Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf (19.69 GB)Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf (23.19 GB)mmproj-BF16.gguf (931 MB) — vision projector (byte-identical to unsloth's)overrides-very-low.txt, overrides-compact-low.txt, overrides-low.txt, overrides-medium.txt, overrides-mid-high.txt, overrides-high.txt, overrides-very-high.txt, overrides-highest.txt — per-tensor quantization maps (1,202 entries each) for reproductionf18098dfc32ca398f63093c113d48bd962684c78a3e93d4158f8f440699f7cae Qwen3.8-27B-NVFP4-MTP-ORIG.gguf
74ea17ea05e0e0241af8d5b29cdea38b3f4509f66d9b96c1ab05f0e1f0e537d9 Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf
ac0ef9c5eceb5a5dc9b266eacc9372508158713c6c35618cf735675451fdd3ac Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf
ce66a629d4a3516bba27ca91de29372f086f90f72ddb92fe298de67b8bb88bbc Qwen3.8-27B-NVFP4-MTP-LOW.gguf
f0b4c538c75037f026bde3b650f0ca639d382c128a4572769dce1183db86253a Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf
79b032f7a118fb34f1445c4d7ae50bc7b304c74161035c3df8bd5526c12899b9 Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf
d57008707b0558bde05ce61d7402e4e668ffd45a4c97d03d2ad97db73f98d403 Qwen3.8-27B-NVFP4-MTP-HIGH.gguf
3e52d6280ee650520a2d901002121c11cf9d23ac75f23c52a362bf285d561d81 Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf
6a202c2faf67f79d4c8c61ec940da7a62bd59a87608508fe8048131630cc4ba6 Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf
83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53 mmproj-BF16.gguf
MMLU letter-logprob (thinking off) and GSM8K-200, identical flags every tier, gate tier of the unified suite. BUDGET and STARVED live in the sibling repo esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUF (no MTP head); the other eight live in this repo.
| Tier | Repo | Size | MMLU (n=1000) | GSM8K (n=200) | PPL |
|---|---|---|---|---|---|
| STARVED | BUDGET | 14.59 GB | 79.56% | 0.915 | 3.7429 |
| BUDGET | BUDGET | 14.72 GB | 80.26% | 0.895 | 3.5689 |
| VERY-LOW | MTP | 14.86 GB | 80.26% | 0.900 | 3.5689 |
| COMPACT-LOW | MTP | 15.16 GB | 79.92% | 0.905 | 3.4875 |
| LOW | MTP | 15.53 GB | 80.75% | 0.920 | 3.5127 |
| MEDIUM | MTP | 16.38 GB | 80.40% | 0.905 | 3.4787 |
| MID-HIGH | MTP | 16.90 GB | 80.84% | 0.915 | 3.5021 |
| HIGH | MTP | 17.57 GB | 80.26% | 0.905 | 3.4784 |
| VERY-HIGH | MTP | 19.69 GB | 78.89% | 0.915 | 3.4924 |
| HIGHEST | MTP | 23.19 GB | 80.46% | 0.905 | 3.4032 |
MMLU sits within sampling noise across the ladder (SE about 1.2pp at n=1000) while perplexity improves monotonically with size.
We did not run a proper benchmark. What follows are informal first impressions from single-stream runs, included only so others know what to expect. Do not treat these as claims.
On reading the numbers: prefill t/s is a mean over the run and degrades as context grows, so figures from different context settings are not directly comparable — the 180k-context runs (~645-650 t/s), the 32k-context runs (~1250-2290 t/s) and the community RTX 5090 numbers all sit at different context sizes, hardware, and prompts. Compare numbers only within the same context and harness.
split-mode: tensor, flash attention on, quantized KV cache.max_tokens 20,000 (repetition_analysis.py). This payload was highly repetitive (the same boilerplate unit ~2,000 times), which likely inflates MTP acceptance and speedups.--spec-type draft-mtp, spec_n_max 6, spec_p_min 0.75.| Tier | Prefill t/s | Decode t/s | n_dec | MTP acc | MTP len | req s |
|---|---|---|---|---|---|---|
| LOW (Q5_0 / IQ4_XS / IQ4_XS) | 645.6 | 18.44 | 1,151 | 0.726 | 2.88 | 361 |
| MEDIUM (Q8_0 / Q6_K / IQ4_XS) | 649.3 | 18.48 | 1,907 | 0.749 | 2.91 | 401 |
| MID-HIGH (Q8_0 / Q8_0 / Q8_0) | 649.0 | 18.00 | 2,465 | 0.765 | 2.82 | 427 |
| HIGH (BF16 / Q6_K / IQ4_XS) | 647.6 | 15.54 | 1,795 | 0.751 | 2.65 | 416 |
| VERY-HIGH (BF16 / BF16 / BF16) | 650.4 | 15.42 | 2,007 | 0.729 | 2.77 | 429 |
Naive first impressions:
The newest tiers were exercised at 32,768 context with quantized q4_0 KV, on the same dual-16GB box, with a 28k-token payload (source-28k.txt) and max_tokens 2000 (COMPACT-LOW ran the same harness at max_tokens 20000).
| Tier | Prefill t/s | Decode t/s | MTP acc | MTP len | req s |
|---|---|---|---|---|---|
| VERY-LOW (Q3_K / Q2_K / Q2_K) | — | — | — | — | not yet benchmarked |
| COMPACT-LOW (Q4_K / Q3_K / Q2_K) | 2281.1 | 25.52 | — | — | 94.8 |
| MID-HIGH (Q8_0 / Q8_0 / Q8_0) | 1451.6 | 25.98 | 0.771 | 2.87 | 108.8 |
| HIGHEST (Q8_0 attn / BF16 embd / BF16 MTP) | 1250.4 | 23.49 | 0.765 | 2.88 | 122.6 |
First impressions:
finish_reason: stop). MTP was left disabled in the deployed config for this run.BUDGET / STARVED ultra-compact tiers in the sibling repo (MTP head stripped to fit 16 GB) hit 2287-2289 t/s prefill and 27.0-27.4 t/s decode at 32k context.A community user benchmarked LOW / MEDIUM / VERY-HIGH on a single RTX 5090 32 GB (9 short prompts, seed 42, --spec-draft-n-max 4, 192K ctx, KV q8_0, temp 1.0). These are their numbers, not ours — see the full gist for the complete results (per-prompt breakdown, method, configs).
| Tier | MTP acc | Avg tok/s | Peak tok/s | VRAM @192K |
|---|---|---|---|---|
| LOW (Q5_0 / IQ4_XS / IQ4_XS) | 49.0% | 148.3 | 203 | 25.0 GiB |
| MEDIUM (Q8_0 / Q6_K / IQ4_XS) | 44.3% | 133.7 | 182 | 25.3 GiB |
| VERY-HIGH (BF16 / BF16 / BF16) | 49.5% | 111.9 | 139 | 26.9 GiB |
| Qwen3.6-27B NVFP4-MTP ref (michaelw9999) | 69.7% | 113.8 | 133 | — |
Their main points:
--spec-draft-n-max sweep (LOW): wall-time peak at n-max 4 (11.5 s / 9 prompts); n-max 2 is the most draft-efficient (68% accepted) but ~10% slower in wall time; n-max 6/8 collapse acceptance (38%/28%) and get slower. Our deployed config uses spec_n_max 6 — on their hardware 4 was fastest; try 1-6 and keep whatever is fastest on yours.ce66a629…, MEDIUM f0b4c538…, VERY-HIGH 3e52d628…) match this repo's current files.Benchmark tool: they published mtp-bench.py (9 prompts, seed 42, accept-rate / tok/s / wall, --long-context, --diff).
llama-server \
--model Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf \
--mmproj mmproj-BF16.gguf \
--ctx-size 262144 \
--flash-attn on \
--spec-type draft-mtp \
--spec-draft-n-max 6 \
--spec-draft-p-min 0.75 \
--temp 0.7 --top-p 0.95 --top-k 20
sm_120 support (Blackwell).draft-mtp spec path (merged upstream as LLAMA_CONTEXT_TYPE_MTP).--mmproj mmproj-BF16.gguf.temp 1.0 / top_p 0.95 / top_k 20; instruct mode temp 0.7 / top_p 0.80 / top_k 20. Pick per use case.--spec-draft-n-max values 1 through 6 and keep whatever is fastest on your system.ORIG file is 33 GB — it fits a single 16 GB GPU only with heavy context reduction or CPU offload. It is included for fidelity and as the tier-derivation source, not as the recommended serving file.Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to Qwen (Alibaba) and Unsloth as above.
"Qwen" is a trademark of Alibaba. Trademarks are used here only to identify upstream models; this repository is not affiliated with, sponsored by, or endorsed by Alibaba or Unsloth.
This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.