An NVFP4 quantization of whittleOrg/Whittle-Next-27B-A3B
(v4.4 weights, qwen4exp architecture), quantized from the official
v4.4-Q8_0 GGUF.
| File | Whittle-Next-27B-A3B-v4.4-NVFP4.gguf |
| Size | 17,094,118,432 B (15.92 GiB) |
| BPW | 4.96 |
general.file_type |
39 (LLAMA_FTYPE_MOSTLY_NVFP4) |
| Source | Whittle-Next-27B-A3B-v4.4-Q8_0.gguf, sha256 79f83ba35d9b0d59d03fd055c85e689862a02adecb36c26401db2f0d53a81d76 |
| Tensors | 1,022 — 674 NVFP4, 324 F32, 20 BF16, 3 Q8_0, 1 F16 |
E2M1 4-bit elements in blocks of 64, with a UE4M3 scale per 16-element sub-block (4 scales per block) — the NVIDIA/OCP microscaling format. That per-16 scaling is why it holds quality better than MXFP4 (one E8M0 scale per 32) at the same bit width.
The body is NVFP4. Three things are deliberately not:
| tensor(s) | type | why |
|---|---|---|
per_layer_token_embd.weight (2B-row n-gram table) |
Q8_0 | It is a RAM read (-ot ...=CPU), not a VRAM cost. Prior measurements on this family say the table must not be traded in either direction. |
token_embd.weight, output.weight |
Q8_0 | Embedding/output are quality-critical per byte. |
31 × *_conv1d.weight |
F16 | ne0 = 4, not divisible by the 64-element NVFP4 block. Forced, not chosen. |
tensor-type-recipe.txt in this repo is the exact --tensor-type-file used.
per_layer_token_embd\.weight=Q8_0
^token_embd\.weight$=Q8_0
^output\.weight$=Q8_0
ple_conv1d\.weight=F16
ssm_conv1d\.weight=F16
Dry run reported 0 fallback quantizations — every targeted tensor got the type asked for.
llama-quantize to reproduce thisStock llama.cpp (checked at 9fba51d) has GGML_TYPE_NVFP4 with full CUDA/Vulkan/CPU
kernels and the llama-graph scale plumbing, but tools/quantize/quantize.cpp has no
NVFP4 entry in QUANT_OPTIONS, so llama-quantize rejects the ftype outright
(invalid ftype 'NVFP4'). Two additions fix it:
// tools/quantize/quantize.cpp, in QUANT_OPTIONS
{ "NVFP4", LLAMA_FTYPE_MOSTLY_NVFP4, " 4.5 bpw NVFP4 (E2M1 + UE4M3 per-16 scales)", },
// src/llama-quant.cpp, in llama_ftype_get_default_type()
case LLAMA_FTYPE_MOSTLY_NVFP4: return GGML_TYPE_NVFP4;
Then quantize from a Q8_0 source with --allow-requantize (required — the source is
already quantized):
llama-quantize --allow-requantize --tensor-type-file recipe.tt \
Whittle-Next-27B-A3B-v4.4-Q8_0.gguf out-NVFP4.gguf NVFP4 24
Cost: 222 s on 24 threads for the 29.3 GB source.
Perplexity, CPU-only, one box (Xeon E5-2670 v3, 24 threads), one corpus, matched flags:
llama-perplexity -m <model> -ot per_layer_token_embd=CPU -ngl 0 -t 24 \
-f corpus.txt -c 4096 -b 512 -ub 128 --chunks 2
corpus: 900,000 B of Gutenberg (Pride & Prejudice + Moby-Dick + Frankenstein, CRLF stripped),
sha256 b2be5aac505abc30bcd91b4b6844ae79b3b245e55d2510cd17b1339dcfcd45f8.
| model | PPL (--chunks 2) |
vs Q8_0 |
|---|---|---|
| Q8_0 source | 8.4790 ± 0.31862 | — |
| NVFP4 | 9.1299 ± 0.35318 | +0.6509 PPL (+7.68%), z = 1.37 |
Per-chunk, so the direction can be checked for consistency rather than trusted once:
| chunk | Q8_0 | NVFP4 | delta |
|---|---|---|---|
| 1 | 12.4937 | 13.6283 | +9.08% |
| 2 | 8.4790 | 9.1299 | +7.68% |
That cell is under-powered (z=1.37) and is kept only for the record. The verdict below is from the tight harness.
--chunks 24, ngl=0, matched corpus / flags / box. Same Xeon E5-2670 v3, 24 threads, ~2.5–3.5 h
per cell so the standard errors are tight enough to decide.
| file | BPW | size | PPL | vs Q4_K_M |
|---|---|---|---|---|
| Q4_K_M (published) | 4.58 | 16.80 GB | 9.7484 ± 0.10485 | — |
| NVFP4 | 4.96 | 17.09 GB | 10.1785 ± 0.11129 | +4.41%, z=+2.81, p=0.005 |
| NVFP4-protect | 5.03 | 17.35 GB | 10.0807 ± 0.10976 | +3.41%, z=+2.19, p=0.029 |
NVFP4 at 4.96 BPW is significantly worse than Q4_K_M at 4.58 BPW — more bits, more bytes,
worse perplexity. The "protect the hybrid tensors" fix (ssm_out/attn_gate → Q8_0,
ssm_alpha/ssm_beta → F16) recovers 0.96% of the gap but that recovery is itself not
significant (z=−0.63, p=0.53) and it still loses to Q4_K_M by a significant +3.41%.
So, for this architecture in stock llama.cpp:
n_ff = 512, the expert tensors are small and shallow; a 64-element FP4
block with 16-element scale granularity quantizes a 512-wide expert projection into 8 blocks,
which is not much of a statistics window.ffn_down_exps is still the tensor that must be protected, as measured on the 45B line —
not the SSM/attention tensors. The next experiment worth running is NVFP4 on gate/up with
ffn_down_exps raised, not more hybrid protection.BLACKWELL_MMA_AVAILABLE, prec_src1 == GGML_PREC_Q4) runs 4-bit activations through tensor cores. All of the above was measured
with 8-bit quantized activations on a CPU — the accuracy story for the intended hardware is a
different question, because activation quantization is where FP4 microscaling is designed to pay.quantize_nvfp4() ignores quant_weights entirely
(GGML_UNUSED(quant_weights) → quantize_row_nvfp4_ref), so unlike k-quants an imatrix
cannot currently improve an NVFP4 quant in llama.cpp at all. That is a real limitation, not a
tuning knob.Those two are why this is "not worth shipping as measured" rather than "NVFP4 is bad". On Blackwell with native W4A4 it may well win on speed at similar quality — but nobody should download this file expecting better tokens than Q4_K_M.
Measured file hashes, for reproducibility:
| file | sha256 |
|---|---|
Whittle-Next-27B-A3B-v4.4-NVFP4.gguf |
7b32c18ff06c2c9c504ed792c8194a717d090673806eb79fcc6fabbd03deb70a |
Whittle-Next-27B-A3B-v4.4-Q8_0.gguf (source) |
79f83ba35d9b0d59d03fd055c85e689862a02adecb36c26401db2f0d53a81d76 |
corpus.txt |
b2be5aac505abc30bcd91b4b6844ae79b3b245e55d2510cd17b1339dcfcd45f8 |
| backend | NVFP4 |
|---|---|
| CUDA | yes — DP4A mmvq on any arch; fast path needs Blackwell MMA (BLACKWELL_MMA_AVAILABLE) |
| Vulkan | yes — dequant/mul_mat_vec shaders; OCP microscaling extension (ocp_fp4) gets the fast path, otherwise a UE4M3 LUT fallback |
| CPU | yes — scalar + x86 SIMD ggml_vec_dot_nvfp4_q8_0 |
| Metal | no |
| ROCm | kernels compile via the CUDA path, but unverified on this file — no AMD card tested |
llama-cli -m Whittle-Next-27B-A3B-v4.4-NVFP4.gguf \
-ot per_layer_token_embd=CPU -ngl 99 -c 16384 --jinja -fa on
-ot per_layer_token_embd=CPU keeps the 2 GB n-gram table in system RAM. It is a cheap
sequential read; leaving it on the GPU wastes VRAM for no accuracy gain.
Base model by logic65 / the Whittle project (Apache-2.0). Quantized and tested by phantomic12.
whittleOrg/Whittle-Next-27B-A3B