Qwen3.8-27B-Opus-Distill-v2-GGUF

GGUF quantizations of barozp/Qwen3.8-27B-Opus-Distill-v2 -- a bugfix release of Qwen3.8-27B-Opus-Distill ("v1").

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset -- every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

Test	v1	v2
"no prose" + "no markdown"	3000/3000 tokens, 0 visible output	87/4096 tokens, clean
"no prose" + "no comments"	3000/3000 tokens, 0 visible output	73/4096 tokens, clean
Generalization (different task)	--	43/4096 tokens, clean

All three converge quickly, </think> closes normally -- no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

Task	Metric	Base	v2	Delta
wikitext	word perplexity (lower better)	8.4335	8.3788	-0.055
mmlu	acc	0.8494	0.8476	-0.002
hellaswag	acc_norm	0.7420	0.7500	+0.008
arc_challenge	acc_norm	0.5880	0.6220	+0.034
gpqa_diamond	acc_norm	0.2323	0.4697	+0.237

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected -- in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) -- this is a same-protocol base-vs-distill delta.

Available quantizations

File	Size	Use case
`Qwen3.8-27B-Opus-Distill-v2-BF16.gguf`	54.7 GB	reference / re-quantization source
`Qwen3.8-27B-Opus-Distill-v2-Q8_0.gguf`	29.0 GB	near-lossless
`Qwen3.8-27B-Opus-Distill-v2-Q6_K.gguf`	22.4 GB	high quality
`Qwen3.8-27B-Opus-Distill-v2-Q5_K_M.gguf`	19.5 GB	quality / balanced
`Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf`	16.8 GB	recommended all-rounder
`Qwen3.8-27B-Opus-Distill-v2-Q3_K_M.gguf`	13.5 GB	tight VRAM
`Qwen3.8-27B-Opus-Distill-v2-IQ3_XXS.gguf`	11.4 GB	low-bit, imatrix
`Qwen3.8-27B-Opus-Distill-v2-IQ2_XXS.gguf`	8.7 GB	very low-bit, imatrix
`Qwen3.8-27B-Opus-Distill-v2-IQ1_M.gguf`	7.9 GB	extreme low-bit, imatrix

K-quants (Q8_0-Q3_K_M) need no imatrix. IQ-quants (IQ3_XXS and below) require one to run at all in current llama.cpp.

Which one to pick:

Best quality with headroom -> Q6_K or Q8_0
Best quality/size balance -> Q4_K_M (default recommendation)
24 GB card -> Q4_K_M; 16 GB card -> Q3_K_M (partial offload)
Below that -> IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo is reused from barozp/Qwen3.8-27B-Opus-Distill-GGUF (v1) -- same base architecture, so no need to recompute. As with v1, the native MTP head (blk.64, the nextn.* decoder layer) is never exercised by a normal forward pass, so it has no imatrix coverage; llama-quantize pins that block to q4_K instead of failing.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf (~0.9 GB). Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

Base: Qwen/Qwen3.8-27B -- dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
Final validation loss: 0.4334
Vision + MTP: carried over byte-for-byte from the base checkpoint -- never trained

Source chain

The full Qwen3.8-27B Opus Distill family:

Qwen/Qwen3.8-27B — base model
barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v2, GGUF quants (the card you are reading).

Thanks to zxbc2023 for the bug report that made this release possible.

barozp/Qwen3.8-27B-Opus-Distill-v2 -- BF16 source model, full card.
barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 -- block-FP8 safetensors build for sglang / vLLM / transformers; includes paired BF16-vs-FP8 benchmark data.
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit and MLX-8bit -- Apple Silicon MLX builds via mlx-vlm.

barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF