Qwen3.8-27B-heretic-ara — GGUF with MTP + Vision

Quantizations of trohrbaugh/Qwen3.8-27B-heretic-ara with MTP (Multi-Token Prediction) tensors preserved (866 tensors) and vision projector (mmproj) for multimodal image understanding.

Includes ROCmFPX quants for the ROCmFPX fork of llama.cpp — ~42 t/s average decode on AMD Strix Halo (50–75% faster than stock Vulkan).

Files

Which quant should I use?

• ROCmFP4_FAST (14 GB): Fastest option on RDNA 3.5 (Strix Halo, gfx1151). Requires ROCmFPX fork with -dev Vulkan0. ~42 t/s average decode. Best choice if you have Strix Halo and want maximum speed.
• Q4_K_M (16 GB): Best for stock llama.cpp on any hardware. ~27–28 t/s on Strix Halo. Full vision + MTP support. Best choice for general use.
• Q6_K (21 GB): Higher quality for creative writing and nuanced reasoning. ~30% slower on unified memory due to larger file size. Choose if quality matters more than speed.
• ROCmFP6 (21 GB): Higher quality ROCmFPX variant. ~24 t/s on Strix Halo. Use if you want both ROCmFPX speed and higher quality, and have memory headroom.

Source model

trohrbaugh/Qwen3.8-27B-heretic-ara — Arbitrary-Rank Ablation (ARA) uncensoring of Qwen3.8-27B using heretic v1.2.0+custom. Parameters: start_layer 26, end_layer 56, preserve_good_behavior_weight 0.9432, steer_bad_behavior_weight 0.0009. KL divergence 0.0535, 0/100 refusals.

How to use

Stock llama.cpp — Text-only with MTP (fastest stock decode)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -ctk f16 -ctv f16 \
  -c 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Swap in Q6_K-MTP.gguf for higher quality. --spec-draft-n-max 3 is optimal on AMD Strix Halo (Vulkan). Try 4 on other hardware.

Stock llama.cpp — Vision + MTP (multimodal)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -c 8192 -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja

Then send images via the OpenAI-compatible API:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
      {"type": "text", "text": "Describe this image."}
    ]
  }],
  "max_tokens": 300
}'

Note: mmproj has zero impact on text-only request performance — the vision encoder only activates for image-bearing requests. Safe to load mmproj permanently.

ROCmFPX — Maximum speed (text-only)

Requires ROCmFPX fork built for your GPU.

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 32768 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6

Critical flags:

• -dev Vulkan0 — forces RADV cooperative matrix path. Without it, ROCm backend runs at ~18 t/s instead of ~42.
• -ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral, dramatically reduces KV memory (enables 262K context in 20 GB vs 61 GB with f16 KV).
• --spec-draft-n-max 6 — optimal for ROCmFPX (vs 3 for stock Vulkan).

ROCmFPX — Vision

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 8192 -np 1 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --jinja

Important: MTP speculative decoding must be disabled for vision requests on ROCmFPX (b213) — the non-consecutive token positions from image embeddings crash MTP boundary tracking. Omit --spec-type draft-mtp when using --mmproj. Vision decode runs at ~13.6 t/s without MTP. Text-only requests will not benefit from MTP in this configuration; use a separate text-only server with MTP for maximum text speed.

Without MTP (slower but works everywhere)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -c 32768

Benchmarks (AMD Ryzen AI Max+ 395, 122 GB unified, gfx1151)

ROCmFPX (Vulkan0, MTP n-max 6, TurboQuant KV)

*ROCmFP6 benchmarked under swap pressure (21 GB model at 8K ctx). Would likely hit ~30+ t/s without swap contention.

Stock Vulkan (MTP n-max 3)

Vision (with mmproj, Q4_K_M, stock Vulkan)

Vision decode is ~13 t/s (vs 28 text-only) due to vision encoder overhead. Text-only requests are not affected by having mmproj loaded.

ROCmFPX vision (without MTP): ~13.6 t/s decode, ~85.9 t/s prefill. MTP must be disabled for vision on ROCmFPX b213.

Stock Vulkan benchmarked on llama.cpp build 0ef6e55ed (b10254). ROCmFPX benchmarked on b213/b2f5829.

Speed comparison summary

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. All quants handle math/reasoning, creative writing, and vision correctly on qualitative tests.

Qualitative comparison

All quants tested on identical prompts at temperature 0.0:

• Math/reasoning: All produce correct step-by-step solutions
• Creative writing: All produce strong prose. Q6 variants show marginally richer vocabulary.
• Vision: All correctly identify shapes, colors, and text in test images.

Architecture

Qwen3.8-27B uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers) with an integrated vision encoder. Key notes:

• Natively multimodal — 333 vision tensors for image understanding (extracted as mmproj)
• Prompt caching is broken in llama.cpp for hybrid-memory models (every turn reprocesses full context)
• Only 16 layers grow KV cache → cheap context scaling
• MTP provides ~2.3× decode speedup via speculative decoding (text-only, stock Vulkan)
• MTP on ROCmFPX provides ~3.2× decode speedup (13 → 42 t/s)

Quantization details

• ROCmFP4_FAST: Q4_0_ROCMFP4_FAST, 4.26 BPW, ~14 GB, 866 tensors (MTP nextn.* → q8_0 auto)
• Q4_K_M: 4.83 BPW, ~16 GB, 866 tensors (851 model + 15 MTP)
• Q6_K: 6.56 BPW, ~21 GB, 866 tensors (851 model + 15 MTP)
• ROCmFP6: Q6_0_ROCMFPX, 6.50 BPW, ~21 GB, 866 tensors (MTP nextn.* → q8_0 auto)
• mmproj: BF16, 931 MB, 334 tensors (vision encoder — CLIP arch, cannot be quantized)
• Source: BF16 safetensors → BF16 GGUF (51 GB) → all quant variants
• Stock quants: convert_hf_to_gguf.py → llama-quantize
• ROCmFPX quants: convert_hf_to_gguf.py → ROCmFPX llama-quantize (b213/b2f5829)
• mmproj: convert_hf_to_gguf.py --mmproj --outtype bf16

MTP tensors included (all quants)

blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.nextn.attn_k.weight
blk.64.nextn.attn_k_norm.weight
blk.64.nextn.attn_norm.weight
blk.64.nextn.attn_output.weight
blk.64.nextn.attn_q.weight
blk.64.nextn.attn_q_norm.weight
blk.64.nextn.attn_v.weight
blk.64.nextn.ffn_down.weight
blk.64.nextn.ffn_gate.weight
blk.64.nextn.ffn_up.weight
blk.64.nextn.ssm_out.weight