Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo

Four files:

M2 — the imatrix quant

second-generation quant: same trunk type (iq4_xs), but calibrated with a 926-entry imatrix (1,024 chunks) via the ROCmFPX banded quantizer, and the 51B PLE lookup table left at q8_0 (no --tensor-type cut). 5.56 bpw, 115 giB — 24 giB bigger than the 91g PLE file.

perplexity (wiki.test.raw, ctx 2048, 145 chunks):

speed profile is honest-mixed (single runs, nathanw1014 vulkan engine, q8_0 kv, temp 0):

pick M2 when you want the best measured quality and don't mind the 24 giB: at ≤32k plain it matches PLE, and at 32k MTP it measured fastest. for shallow-depth MTP speed take the 91g PLE; for deep 128k+ MTP the static 116g had a small in-sweep edge (17.4 vs 13.3/13.5 — within the same-config spread, n=1 caveat).

provenance

every quant descends from an F16 that was byte-verified against the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint: hyper-connection norms folded to (1 + w) (97/97 tensors — the converter bug that produces deterministic garbage is fixed in our pipeline), PLE fp8 scale applied, expert stacking identity probed 512x3, GDN v-head reorder checked. details: https://github.com/julianmb/haloq38flash

the PLE cut (what makes the 91G special)

the 51B-parameter n-gram lookup table was moved from q8_0 (54G) to iq4_nl (27G) via --tensor-type. hash-gathered lookup rows tolerate the precision drop — verified by smoke and full benchmark, no degradation observed:

prefill at 32k: 384 → 397 t/s. no collapse at depth. engine: nathanw1014 strix-halo-vulkan (ad914eb), vulkan/radv, q8_0 KV, -ub 2048, temp 0.

fork compatibility caveat (important)

the iq4_nl PLE rows assert in SOME forks: engines that feed gathered PLE rows directly as mul_mat B operands without dequantizing (apepojken qwen4exp-spec-mtp) abort at ggml-vulkan.cpp:7794 (b_type must be F32/F16/Q8_1). verified working on nathanw1014 strix-halo-vulkan. if your engine asserts on load or first token, use the 116G static file instead. M2 keeps the PLE table at q8_0 and has no such assert exposure.

provenance note

the same --tensor-type cut applied to unverified-source quants will NOT fix a broken converter (hc norms, PLE scale) — garbage in, garbage out. ours is built from a fixed, audited pipeline.

128k & 256k context benchmarks (SSD-PLE)

the depth story is not monotonic. measured on the same engine (nathanw1014 vulkan, q8_0 kv, temp 0):

• 256k context unlocked with SSD-PLE: With -lm mmap --tensor-read-lazy on, the 27 GB PLE table is kept on NVMe SSD, keeping active RAM under 95 GB and providing ~30 GB of free headroom at full 256k context. Combined with -tb 16 and -ub 1024 -b 2048 -t 4, this achieves 15.2 t/s MTP generation and 179-191.5 t/s prefill at 256k with zero swap thrashing.
• Hardware boundary: Do not use -ub 2048 at 256k context; it triggers Vulkan queue submission timeouts (ErrorDeviceLost). -ub 1024 is the tested maximum.
• 128k reversal: At 128k specifically under MTP, the static 116g quant has higher draft acceptance than the 91g PLE quant (26.9 vs 18.6 t/s).
• Practical: Use the 91g PLE file for daily chat, coding, and full 256k long context (with SSD-PLE flags). Use the 116g static file if you specifically need highest MTP speed at 128k.

run it (strix halo, 128 GB unified memory)

full methodology, receipts, and the benchmark record: https://github.com/julianmb/haloq38flash

docker (one-liner; image default serves the 91g PLE on :8080):

git clone https://github.com/julianmb/haloq38flash && cd haloq38flash
docker compose up --build

point it at M2 with the MTP sidecar and the warm-turn cache:

docker compose run qwen38-flash-next /app/llama-server \
  -m /models/Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md /models/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --cache-ram 8192 --ctx-checkpoints 32 \
  -c 32768 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4

or raw llama.cpp (nathanw1014 strix-halo-vulkan lineage engines):

llama-server -m Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4 -c 32768

perf notes: -t 16 lifts prefill up to +43% at 128k (decode indifferent); the --cache-ram/--ctx-checkpoints warm-turn flags make repeated context nearly free (measured 438 s → 0.68 s at 128k, 994 s → 0.74 s at 256k — 640x/1351x). swap M2's filename for the other quants; same flags.

license

qwen community license 1.0 (distribution permitted with notice; maas restrictions apply). base model: Qwen/Qwen3.8-Flash-Next.