Qwen3.8-27B-Opus-Distill-v2-MLX-4bit

MLX 4-bit conversion of barozp/Qwen3.8-27B-Opus-Distill-v2 -- the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") -- for Apple Silicon. Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, the FP8 build for CUDA serving stacks (sglang/vLLM), and the 8-bit MLX repo.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset -- every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

All three converge quickly, </think> closes normally -- no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected -- in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) -- this is a same-protocol base-vs-distill delta.

Available MLX conversions

Which one to pick:

• Best quality with headroom -> 8-bit
• Best size/speed balance -> 4-bit (default recommendation)
• 24 GB Mac -> 4-bit (cap large images with --max-pixels, see Vision); 36 GB+ Mac -> either
• On llama.cpp/Ollama/LM Studio workflows -> use the GGUF repo instead

Conversion notes

• Command: python -m mlx_vlm.convert --hf-path <source> --mlx-path <out> --quantize --q-bits 4 --q-group-size 64
• Tooling: mlx 0.32.1 · mlx-vlm 0.6.8 · huggingface_hub 1.27.0; source commit 75e188fcd327
• Output: 2180 tensors, 3 safetensors shards, quantization mode affine; the full conversion log ships in this repo as conversion_log.txt
• Environment: Google Colab G4 (RTX PRO 6000 Blackwell 96 GB), MLX CUDA backend
• As with the GGUF builds, the native MTP head (blk.64, the nextn.* decoder layer) is not usable here either -- mlx-vlm's loader strips mtp.* weights during conversion, so these checkpoints do not offer native-MTP self-speculative decoding

Vision

Unlike the GGUF builds (where the tower ships separately as mmproj), the vision encoder is inside these weights -- image input works out of the box, no extra file to load:

# multimodal generation
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit \
  --prompt "Describe this image." \
  --image photo.jpg

Text-only usage does not need anything extra and runs fine without --image. High-resolution photos expand into many visual tokens (~12k observed); on unified-memory Macs, cap resolution when needed with --max-pixels 1003520.

Quick start

# install
pip install -U mlx-vlm

# text-only chat
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --max-tokens 512

# multimodal
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit \
  --prompt "Describe this image." \
  --image photo.jpg

# OpenAI-compatible local server
python -m mlx_vlm.server \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B -- dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
• Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
• Final validation loss: 0.4334
• Vision + MTP: carried over byte-for-byte from the base checkpoint -- never trained

Source chain

The full Qwen3.8-27B Opus Distill family:

• Qwen/Qwen3.8-27B — base model
• barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
• barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v2, MLX 4-bit (the card you are reading).

Thanks to zxbc2023 for the bug report that made this release possible.