Qwen3.8-27B Heretic ARA ModelOpt NVFP4 GGUF

Text-only GGUF conversion of PassingByPixels/Qwen3.8-27B-heretic-ara-NVFP4, paired with a compact Q4_K_M MTP-only draft model.

Lineage

Qwen/Qwen3.8-27B
  -> trohrbaugh/Qwen3.8-27B-heretic-ara
  -> PassingByPixels/Qwen3.8-27B-heretic-ara-NVFP4
  -> this GGUF conversion

• Heretic ARA source: 1-pass ARA checkpoint, reported KL divergence 0.0535.
• Quantization: NVIDIA ModelOpt NVFP4 W4A4, group size 16.
• Conversion: official llama.cpp b10453, --outtype auto --no-nextn.
• GGUF payload: 1,651 tensors; 400 native NVFP4 tensors (GGML_TYPE_NVFP4).
• This is not the RVN 3-pass checkpoint and does not claim RVN's reported KL value.
• No Q4/K-quant checkpoint was used as a quantization source.

Files

The MTP file is derived from the original Qwen3.8-27B MTP head and is compatible with Qwen3.8-27B derivatives. It is not trained on the Heretic weights.

llama.cpp

Requires a recent CUDA build with Qwen3.8, NVFP4, and draft-MTP support. Native FP4 execution requires NVIDIA Blackwell.

llama-server \
  -m Qwen3.8-27B-Heretic-ARA-ModelOpt-NVFP4.gguf \
  -c 262144 -np 1 --kv-unified -ngl all -fa on \
  -ctk q8_0 -ctv q4_0 \
  --spec-type draft-mtp \
  --spec-draft-model Qwen3.8-27B-MTP-ONLY-Q4_K_M.gguf \
  --spec-draft-n-max 3 -ctkd q4_0 -ctvd q4_0 \
  -b 512 -ub 256 -fit off \
  --jinja --reasoning on --reasoning-preserve

Disable thinking per request:

{"chat_template_kwargs":{"enable_thinking":false}}

RTX 5090 validation

Tested on one GeForce RTX 5090 32 GB with llama.cpp b10453, CUDA 13.3 runtime, 262,144 allocated context, one slot, Flash Attention, K Q8_0 / V Q4_0 target KV, and Q4_0 draft KV.

• Load VRAM: approximately 27.22 GB.
• Post-request VRAM: approximately 27.25 GB.
• A 9,476-token retrieval prompt completed without OOM: 68.17 prompt tok/s and 21.09 generation tok/s.
• Draft maxima 3, 5, and 7 were tested. 3 was retained because 5 and 7 reduced throughput on low-acceptance prose.
• enable_thinking=false, Korean and English output, coding, authentication, start/stop, and VRAM release were tested through the OpenAI-compatible API.

These are single-system smoke measurements, not general benchmark claims. Throughput depends heavily on prompt structure, MTP acceptance, context length, runtime revision, and GPU state.

Sampling defaults used in validation

temperature       0.6
top_p             0.95
top_k             20
min_p             0.0
presence_penalty  0.0
repeat_penalty    1.0

Limitations

• Reduced safety guardrails are inherited from the upstream Heretic ARA checkpoint.
• This release is text-only. Vision and video paths were not included or validated.
• The runtime KV cache in the documented llama.cpp configuration is Q8_0/Q4_0, not FP8.
• Allocating 262K context does not make full-window prefill fast. Long-context attention cost remains substantial.
• Quantization is lossy. No full academic evaluation suite was run for this GGUF conversion.

Credits

• Qwen — base model.
• trohrbaugh — Heretic ARA checkpoint.
• PassingByPixels — NVIDIA ModelOpt NVFP4 checkpoint.
• a4lg — MTP-only GGUF conversion.
• llama.cpp — GGUF conversion and runtime.

License

Apache License 2.0, inherited from the upstream model lineage.