Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit

MTPLX 4-bit conversion of barozp/Qwen3.8-27B-Opus-Distill-v2 — the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") — for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, the 3-bit MTPLX repo, and MLX-4bit / MLX-8bit for non-speculative MLX.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

Same bugfix as the 3-bit MTPLX sibling: v1's training data included reconstructed reasoning traces that induced a deterministic self-verification loop on stacked format constraints ("no prose" + "no markdown" → 3000/3000 tokens, zero visible output). This release retrains on a rebuilt dataset — every row traced to a verified genuine Opus source, 30-way parallel LLM-judge scan, pathological rows dropped.

The 4-bit MTPLX point is the best quality/size balance in the family: ~16.9 GB on disk, vision intact, and the highest verified speculative speedup in this collection (2.39x at depth 3). If you have the headroom, pick this over the 3-bit.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

All three converge quickly, </think> closes normally — no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1.

General knowledge and language modeling are essentially unchanged; reasoning shows a large same-protocol gain over base.

Available MTPLX conversions

Which one to pick:

• Best quality + highest speculative throughput → this 4-bit MTPLX (2.39x at D3)
• Tightest fit / 24 GB Mac with large images → 3-bit MTPLX (1.63x at D2)
• No speculation → MLX-4bit; llama.cpp → GGUF Q4_K_M / Q6_K

Conversion notes

• Source: barozp/Qwen3.8-27B-Opus-Distill-v2 (bf16, barozp--Qwen3.8-27B-Opus-Distill-v2 local cache), forge-local via mtplx==2.9.1
• Recipe: body_bits=4, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, quantized trunk + bf16 MTP sidecar
• MTP contract: base_hidden_variant=post_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
• Output: 3 safetensors shards + model-vision.safetensors (879 MB, 333 tensors, bf16) + mtp.safetensors (810 MB, bf16 sidecar) + model.safetensors.index.json
• Forged at: 2026-08-24T14:12:22+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) — tuned for 24 GB Macs · macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Quality-gate nuance: an earlier 2048-token-budget verify run for this 4-bit checkpoint was flagged quality_failed (strict judge, long-form drift). The current in-repo verification uses the standard 512-token budget — the same budget used for every other MTPLX card in this collection — and passes cleanly at all depths. No weights were changed between runs; only the verify budget was normalized. The table below is the normalized 512-token result.

Vision

The vision encoder is inside these weights (stored as model-vision.safetensors, auto-loaded) — no separate mmproj to fetch:

mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Describe this image." --image photo.jpg --depth 3

Text-only needs nothing extra. Cap max-pixels if you hit unified-memory pressure on very high-res images.

MTPLX usage

# install
pip install -U mtplx

# text-only chat (verified optimum is --depth 3 for this 4-bit)
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 3 --max-tokens 512

# multimodal
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Describe this image." --image photo.jpg --depth 3

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --depth 3 --port 8080

Defaults verified: temperature=0.6, top_p=0.95, top_k=20, recommended_profile=sustained.

Verification results

Verified locally with mtplx forge verify (MLX backend, 512-token budget, quality gate enabled). Depth 0 is plain AR.

• Recommended depth: 3 (mtp_depth_wins at D3)
• Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
• Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) · macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:06854b44...
• All depths passed the quality gate; D3 is both the fastest and the deepest verified depth for this checkpoint.

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
• Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
• Final validation loss: 0.4334
• Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Source chain

Qwen/Qwen3.8-27B (base) → barozp/Qwen3.8-27B-Opus-Distill-v2 (LoRA finetune, safetensors) → this repo (MTPLX 4-bit conversion)

Thanks to zxbc2023 for the bug report that made this release possible.