leonsarmiento/Qwen3.8-27B-pi-4bit-mtp-mlx

⚡ oMLX REQUIRED: These quants are designed to be used with oMLX and they will not work on LM studio. The bundled MTP (Lightning MTP) speculative-decoding heads only execute on oMLX — LM Studio loads the base model but ignores the MTP heads, so the acceleration is lost. For MTP speed, serve this model with oMLX.

Uniform 4-bit MLX quantization of bytkim/Qwen3.8-27B-pi with the model's own Multi-Token Prediction heads baked in — same bundled-mtp.* checkpoint format as Jundot/Qwen3.8-27B-oQ4e-mtp, so oMLX's Lightning MTP speculative decoding engages natively. No separate drafter to download or pair.

About Qwen3.8-27B-pi

A coding-specialized fine-tune of Qwen3.8-27B built for the Pi agent harness — the everyday loop of reading a repository, editing files, running tools, and responding to feedback. Trained via supervised fine-tuning on filtered, successful Pi coding sessions, then refined with GRPO reinforcement learning using a custom reasoning-efficiency reward (more completed, checked work with less generated text).

Per the source benchmarks: Pi's medium reasoning effort matches the base model's xhigh completion rate with ~41% fewer output tokens. Full details: bytkim/Qwen3.8-27B-pi.

Quantization Details

• All quantized layers (attention, linear attention, MLP, embeddings, lm_head, vision tower): uniform 4-bit affine, group_size=64
• Norms: bf16
• Vision tower: preserved (multimodal via mlx-vlm)
• Chat template: byte-identical to source (and to the Jundot oQ4e-mtp donor), reasoning_effort defaulting to xhigh

Bundled MTP (extracted from the source's mtp.safetensors, scottlowry/oQ checkpoint convention — identical key layout to the Jundot donor):

• 7 norm tensors: bf16, +1.0 MLX RMSNorm convention
• fc.weight: bf16 unquantized (bit-identical to source)
• 7 linear layers: 4-bit affine, group_size=64 (covered by the global 4-bit default — no per-tensor overrides needed)
• text_config.mtp_num_hidden_layers: 1, mtp_use_dedicated_embeddings: false

Stats: 15.92 GB · 3 shards · 4.558 bpw · mlx-vlm · vision preserved. Uniform 4-bit vs the donor's data-aware oQ4e (4-bit + 166 five-bit linear-attention overrides): this build is calibration-data-free, while oQ4e tilts precision toward its calibration domains.

Benchmarks (oMLX on an M4 Pro Mac (48 GB), Engine Auto, Context: Code/Python, TurboQuant off)

Same-session back-to-back runs, oMLX bench pp1024/tg128:

• Prefill parity with the data-aware donor (slightly ahead) at 1.05 GB less disk — uniform 4-bit is lighter than oQ4e's 4+5-bit mix.
• Decode within ~6% of the donor (draft acceptance 84–85% vs 86% on coding prompts; pi's own MTP heads quantized to plain 4-bit). Longer 256-token runs measured from oMLX's cycle logs: 30.8 vs 32.0 tok/s steady — same gap.
• MTP-off baseline: ~14.9 tok/s. Fresh oMLX model entries default mtp_enabled: false — if decode sits at ~15 tok/s, enable MTP in the model settings (TurboQuant KV stays off) and reload; verify MTP path activated in ~/.omlx/logs/server.log.

Usage

Serve with oMLX and keep TurboQuant KV cache disabled — MTP shows no advantage with it enabled.

Reasoning parsing / tool calling (from the source model): reasoning_parser: qwen3, tool_call_parser: qwen3_xml.

Sampling — thinking mode (source defaults): temperature 1.0, top_p 0.95, top_k 20.

Sampling — non-thinking mode (Qwen recommended): temperature 0.7, top_p 0.8, presence_penalty 1.5, with enable_thinking: false.

Reasoning effort: the chat template accepts reasoning_effort ∈ {low, medium, high, xhigh} (default xhigh). Pi's RL makes low/medium unusually token-efficient — worth trying for routine edits.

When to turn MTP off: thinking/reasoning-heavy tasks. Faster token generation gets spent on more reasoning, not faster answers (measured on Qwen3.6-35B MATHQA: −7pp accuracy with MTP on, +20% wall time).

Source

bytkim/Qwen3.8-27B-pi — built on Qwen/Qwen3.8-27B: dense hybrid-attention VLM, 64 layers (48 linear attention + 16 full attention), 262K context, MTP prediction layer, vision tower.

Acknowledgements

Thanks to bytkim for Qwen3.8-27B-pi and to Jundot for the bundled-MTP checkpoint format this build replicates.