Qwen3.8-27B — ROCmFPX quants (Q8 full + 16 GB hybrid)

These are quantizations of unsloth/Qwen3.8-27B-GGUF (BF16) (original model: Qwen/Qwen3.8-27B, the hybrid attention/SSM qwen35 architecture with an MTP nextn head).

[!IMPORTANT] You need the ROCmFPX fork of llama.cpp (or a llama.cpp build with ROCmFPX support). These files use the experimental q8_0_rocmfpx (type 103) and/or q4_0_rocmfp4_fast (type 101) weight formats, which stock llama.cpp releases do not understand — loading them elsewhere will fail with an unknown tensor type error.

Files

File	Recipe	Size	For
`Qwen3.8-27B-Q8_0_ROCMFPX.gguf`	pure `q8_0_rocmfpx`, all weights	26.9 GB (8.25 bpw)	large-VRAM systems (e.g. Strix Halo)
`Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf`	bulk `q4_0_rocmfp4_fast` + sensitive tensors at `q8_0_rocmfpx`	16.4 GB (5.15 bpw)	~24 GB VRAM laptops (leaves room for KV cache)

Hybrid recipe (16 GB file)

Sensitive-tensor selection mirrors the tiers in Unsloth's UD-Q4_K_XL dynamic recipe (Q6_K tier = most sensitive, Q5_K = next), rebalanced onto a strict two-type q4/q8 mix to hit ~16 GB:

Tensors at `q8_0_rocmfpx` (165 tensors)	Why
`attn_q/k/v/output` (17 full-attention layers)	attention projections (Unsloth Q5_K/Q6_K tier; `attn_v` is Q6_K there)
`attn_gate` + `ssm_out` (48 linear-attn/SSM layers)	Q5_K tier in UD-Q4_K_XL
`output.weight` head	Q6_K tier in UD-Q4_K_XL

Everything else is q4_0_rocmfp4_fast (340 tensors, incl. ffn_gate/up/down, attn_qkv, token_embd — Unsloth keeps embeddings at Q4_K too), norms/biases stay f32 (360 tensors). The MTP head (nextn.eh_proj) is auto-protected at q8_0 by the quantizer's draft-sensitive logic.

How they were made

# pure Q8 (from the ROCmFPX fork; CPU-only build works fine for quantization)
llama-quantize Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q8_0_ROCMFPX.gguf Q8_0_ROCMFPX

# 16 GB hybrid
llama-quantize \
  --tensor-type "attn_q.weight=q8_0_rocmfpx" \
  --tensor-type "attn_k.weight=q8_0_rocmfpx" \
  --tensor-type "attn_v.weight=q8_0_rocmfpx" \
  --tensor-type "attn_output.weight=q8_0_rocmfpx" \
  --tensor-type "attn_gate.weight=q8_0_rocmfpx" \
  --tensor-type "ssm_out.weight=q8_0_rocmfpx" \
  --tensor-type "^output.weight=q8_0_rocmfpx" \
  Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf Q4_0_ROCMFP4_FAST

Usage

This quant is text weights only. Qwen3.8 is multimodal — for vision support, pair it with the mmproj file: this repo includes mmproj-F16.gguf (mirrored from unsloth/Qwen3.8-27B-GGUF, which also offers mmproj-BF16.gguf).

# build ROCmFPX for your GPU (see the repo README; e.g. Strix Halo):
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh

./build-strix-rocmfp4/bin/llama-cli \
  -m Qwen3.8-27B-Q8_0_ROCMFPX.gguf --mmproj mmproj-F16.gguf \
  -dev Vulkan0 -ngl 999 -fa on --jinja

Benchmarks

[!NOTE] All Benchmarks were run on an AI max 395+ with 128GB RAM

Model	Size	Depth	pp512 t/s	tg128 t/s
Q8_0_ROCMFPX	26.25 GiB	4096	304.65 ± 1.28	7.47 ± 0.03
Q8_0_ROCMFPX	26.25 GiB	16384	247.26 ± 1.00	7.37 ± 0.10
Q8_0_ROCMFPX	26.25 GiB	65536	143.50 ± 0.43	6.84 ± 0.00
Q4_0_ROCMFP4_FAST	16.38 GiB	4096	329.88 ± 4.00	11.15 ± 0.13
Q4_0_ROCMFP4_FAST	16.38 GiB	16384	262.43 ± 1.17	10.95 ± 0.02
Q4_0_ROCMFP4_FAST	16.38 GiB	65536	147.77 ± 0.59	9.49 ± 0.02

Quality comparison vs BF16 source (perplexity / KLD): TBD.

Attribution & license

Quantized from: unsloth/Qwen3.8-27B-GGUF (BF16 shards); sensitivity tiers referenced from their UD-Q4_K_XL
Original model: Qwen/Qwen3.8-27B
License: Apache-2.0 (inherited)
Quant formats by the ROCmFPX project