Qwen3.6-35B-A3B — APEX i-quality-torch vs NVFP4

TLDR: 31 July 2026 NVFP4 support in llama.cpp doesn't deliver on the GB10 yet. This model required patching the converter and then ran ~44% slower than my APEX quant for identical bit size. It did load faster, but that matters almost not at all for my applications.

A head-to-head between a hand-tuned APEX depth-band quant and unsloth's NVFP4-Fast quant of the same base model — same architecture, nearly identical overall bit budget, very different allocation philosophy. NVFP4 is NVIDIA's 4-bit block-scaled float format.

Models compared

Both start from the same Qwen/Qwen3.6-35B-A3B checkpoint (hybrid linear-attention + latent-MoE with a shared expert, native MTP head).

Scope note: plain unsloth/Qwen3.6-35B-A3B-NVFP4 and nvidia/Qwen3.6-35B-A3B-NVFP4 were originally planned additions to this comparison, but that work was dropped once the NVFP4-Fast results below made clear that the interesting variable here is llama.cpp's NVFP4 decode kernel maturity, not producer-to-producer differences — see the callout in Results. All NVFP4 checkpoints from any producer would hit the same kernel-level decode ceiling on this hardware, so a 4-way comparison wouldn't add new signal over the 2-way one below.

Note on MTP: the NVFP4-Fast checkpoint does not retain the model's native multi-token-prediction (MTP/nextn) tensors — unsloth's config.json explicitly lists re:^mtp.* in its ignore list. The APEX torch quant keeps MTP (Q6_K), enabling free --spec-draft-n-max 2 self-speculative decoding that the NVFP4 quant can't use in this form. Speed numbers below are reported both with and without speculative decoding so the comparison is apples-to-apples on the quant format itself, with the MTP bonus called out separately.

Structural bit allocation: APEX vs NVFP4-Fast

Per-tensor-role comparison, read directly off each checkpoint's tensor shapes/dtypes (not just aggregate bpw):

The two schemes converge on the same instinct — protect attention/router, compress the big expert FFNs — but differ sharply in how. Unsloth's NVFP4-Fast spends a noticeably higher budget on attention (FP8 vs APEX's Q6_K) but pays for it by being flat and uniformly aggressive on every expert tensor (~4-bit NVFP4, no depth differentiation). APEX spends less on attention and uses the savings to differentiate within the experts — edge/near-layer experts stay at Q5_K/Q6_K while only the coldest middle-depth layers drop to IQ4_XS. Same rough bit budget, opposite allocation philosophy: NVFP4 is format-uniform, APEX is depth-aware.

Measured result: APEX wins on both axes. At a slightly smaller file size (23.51 GB vs 24.58 GB, 5.298 vs 5.539 bpw), APEX's depth-aware allocation gets a lower (better) perplexity than the flat NVFP4 allocation — see Results below. Spending the bit budget where the model's own structure says it matters (depth-band-differentiated experts) outperforms spending it uniformly, even though NVFP4-Fast's attention tensors get a strictly higher-precision format (FP8 vs Q6_K).

Results

APEX wins smaller, better, and faster (where it counts): 1.07 GB smaller, better perplexity, and 47% faster decode — the one place NVFP4-Fast leads is prefill (+16%), which matters less for interactive/agentic workloads than decode does.

Quality: APEX wins. 1.07 GB smaller, and perplexity is lower (better) by 0.16 — well outside the combined error bars (6.9623 ± 0.046 vs 7.1236 ± 0.048). Depth-aware allocation is beating uniform 4-bit-everywhere-in-the-experts at this bit budget.

Speed: it splits by phase, and the split is not what bpw alone would predict. NVFP4-Fast has faster prompt processing (2475 vs 2127 t/s, +16%), but APEX has substantially faster decode (71.06 vs 48.27 t/s, +47%) — the more practically important number for interactive/agentic use, since that workload is decode-dominated, not prefill-dominated. (The MTP n=2 row above is a separate, not-yet-measured number — the ~45 tok/s median seen in the agentic coding benchmark is a different metric, output-tokens-over-full-agentic-wall-clock including tool calls and repeated prompt reprocessing, and isn't directly comparable to this decode-only figure.)

This is llama.cpp's current NVFP4 kernel, not a property of the NVFP4 format. Its prefill-favoring, decode-flat-to-regressing pattern is documented upstream across multiple PRs on multiple Blackwell GPUs — PR #21074 (generic MMQ kernel, RTX 5090): prefill up 1.27x–3.89x, decode flat at 0.99x–1.00x, on every tested model; PR #22196 (Blackwell tensor-core path, GB10 benchmark on Nemotron-3-Super-120B): prefill +20.8%, decode −0.8%. The kernel author states it directly: "I struggled to find a good balance using the existing generic kernels... no other combination tried gave as significant prefill increases without also reducing tg [decode]." A genuine Blackwell MMA kernel to close the decode gap is flagged as future work, not yet landed. Maintainers separately note decode is memory-bandwidth-bound and NVFP4 should decode at roughly parity with same-size Q4_K in principle (discussion #23627) — its real edge is meant to be the prefill/compute-bound case, from skipping a dequant step via FP4 tensor cores.

Our measured gap (47% slower decode) is larger than the ~1% seen in those reference benchmarks, plausibly because Qwen3.6-35B-A3B is a 3B-active MoE at batch=1 — many small per-expert matmuls, exactly the low-occupancy/high-register-pressure regime the kernel author flagged as the cost of the prefill optimization. Roofline sanity check: GB10's real memory bandwidth is ~273 GB/s; at ~2 GB of active weight read per token (3B active × ~5.5 bpw), that implies a ~130 tok/s ceiling. The k-quant result (71 tok/s) reaches ~55% of that roofline — normal for llama.cpp; NVFP4 (48 tok/s) reaches ~37% — consistent with genuine extra per-token kernel overhead on this architecture, not measurement noise.

PPL methodology: llama-perplexity against wiki.test.raw, 512-token chunks, same corpus/chunking for every quant. Speed methodology: llama-bench -p 512 -n 128, no speculative decoding, measured directly on this hardware (DGX Spark GB10, aarch64, Blackwell — llama.cpp has native NVFP4 CUDA kernels with a Blackwell-specific tuning config, so these are real GPU-kernel numbers, not a CPU-dequant fallback).

Reproduce

# Convert an NVFP4 (compressed-tensors, mixed-precision) checkpoint to GGUF.
# Requires a llama.cpp build with the two conversion fixes below.
python3 convert_hf_to_gguf.py <local-hf-checkpoint-dir> \
  --outfile <name>.gguf --outtype auto

# PPL
llama-perplexity -m <name>.gguf -f wiki.test.raw -c 512

# Speed (no speculative decoding — format-only comparison)
llama-bench -m <name>.gguf -p 512 -n 128

# Speed (APEX torch quant, with native MTP self-speculation)
llama-server -m Qwen3.6-35B-A3B-APEX-i-quality-torch.gguf \
  --ctx-size 8192 --spec-type draft-mtp --spec-draft-n-max 2

Converter fixes required for mixed-precision compressed-tensors NVFP4

conversion/base.py (llama.cpp) only handled a single compressed-tensors config group. Two fixes were needed to support genuine mixed-precision checkpoints (one FP8 group + one NVFP4 group, as used by unsloth's Qwen3.6 NVFP4 releases):

1. Multi-group detection: the NVFP4-checkpoint detection used all(...) over config groups where any(...) was correct, and the non-NVFP4 group (FP8 attention/lm_head) had no dequant path at all — added one that reads the FP8 group's config and dequantizes it the same way the single-group float-quantized case already does.
2. FP8/NVFP4 tensor misrouting: lm_head's FP8 per-channel scale has shape [out_features, 1] — technically 2D, which is indistinguishable from a genuine (if degenerate) single-block NVFP4 scale by ndim alone. This routed lm_head into the NVFP4 repack path and crashed on a shape mismatch. Fixed by additionally requiring scale.shape[-1] > 1 (a real per-block NVFP4 scale always covers more than one block per row; FP8's per-channel scale never does).

Attribution & license

• Base model: Qwen/Qwen3.6-35B-A3B, Apache-2.0.
• NVFP4 quant: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast — see its model card for license terms.
• APEX quant + this comparison: MIT, this repo.