ToPo-ToPo/Qwen3.8-27B-MTP-4bit
MTP (multi-token prediction) drafter for speculative decoding with
Qwen/Qwen3.8-27B on Apple Silicon (mlx-vlm).
This is not a standalone chat model — it only works bound to a Qwen3.8-27B target model.
Provenance
- Split from the official checkpoint's built-in
mtp.*weights (15 tensors, isolated inmodel-00018-of-00018.safetensors) with mlx-vlm 0.6.9:python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split --model Qwen/Qwen3.8-27B --output . - Precision: affine 4bit / group 64 — 4.501 bits/weight
- Size: 247 MB,
block_size: 3,model_type: qwen3_5_mtp
Speculative decoding notes
Gains are largest on structured output (code, JSON tool calls) and smallest on free-form prose. Stable over 589 rounds with rejections — no rollback crash.
The bf16 and 4bit drafters accept identically under greedy decoding (quantization rarely flips the argmax); bf16 was marginally better on code, 4bit uses less memory.
Measure it yourself. Speedup depends on hardware, target quantization, workload and the
mlx-vlm version — on the machine used here the reference pair
Qwen3.6-27B-4bit + mlx-community/Qwen3.6-27B-MTP-4bit showed no speedup at all, so any
number quoted elsewhere may not transfer to your setup.
Usage
mlx_vlm.generate --model ToPo-ToPo/Qwen3.8-27B-mlx-4bit \
--draft-model ToPo-ToPo/Qwen3.8-27B-MTP-4bit --draft-kind mtp \
--prompt "..." --max-tokens 400