Sign up

yc777/Qwen3.8-27B-MTPLX-Optimized-Speed-M3Max-Q4G64

🤗 Hugging Face sourcetext-generationapache-2.026.9B params54 GBsafetensors✓ HF checksums availableupdated today
No torrent yet →

Qwen3.8-27B MTPLX Optimized Speed — M3 Max Q4 G64

An MLX 4-bit affine group-size-64 quantization of the official Qwen/Qwen3.8-27B weights, packaged for MTPLX 2.6.0 native MTP inference on Apple Silicon.

What is included

  • Quantized trunk: 4-bit affine, group size 64 (model-00001...00003.safetensors).
  • Native MTP sidecar: mtp.safetensors, intentionally retained in BF16.
  • MTP runtime contract: mtplx_runtime.json, architecture qwen3-next-mtp.
  • Tokenizer, chat template, generation config and model index.

The MTP sidecar is required. Do not delete it or re-quantize it as a generic MLX artifact if you want native speculative decoding.

Provenance and license

  • Base model: Qwen/Qwen3.8-27B.
  • Source format: official BF16 native weights.
  • Conversion: MTPLX Forge 2.6.0, performed locally on 2026-08-15.
  • Quantization recipe: body Q4, affine, group size 64; MTP policy keep_bf16.
  • License: Apache-2.0. This derivative must retain the upstream LICENSE and attribution.

This is an independently produced quantization, not an official Qwen or MTPLX release.

Requirements

  • Apple Silicon Mac with sufficient unified memory; this package occupies about 15 GB on disk.
  • MTPLX 2.6.0 or newer with qwen3-next-mtp support.
  • This package is intended for MTPLX, not oMLX 0.5.3 direct loading. That oMLX version does not support this native MTP package.

Run

mtplx quickstart \
  --model /path/to/Qwen3.8-27B-MTPLX-Optimized-Speed-M3Max-Q4G64 \
  --host 127.0.0.1 --port 18001 \
  --profile sustained \
  --mtp --depth 3 \
  --scheduler-mode serial --batching-preset solo \
  --reasoning off --preserve-thinking off \
  --max-tokens 8192 --no-stats-footer --yes

The service is OpenAI-compatible at http://127.0.0.1:18001/v1. Confirm that response mtplx_stats includes generation_mode: "mtp", runtime_mtp_enabled: true and draft_head_installed: true.

Validation notes

Forge contract validation recognized qwen3-next-mtp and passed quality checks. Its depth-2 validation sample measured 21.99 tok/s versus an 18.53 tok/s autoregressive baseline (1.187x). This is a conversion/quality-validation figure, not the recommended serving benchmark: the Forge harness uses a specific prompt, a conservative comparison setup and does not represent the final depth-3 MTPLX service profile.

On the M3 Max 128 GB deployment that produced this artifact, MTPLX 2.6.0 sustained with native MTP depth 3 completed a 512-token API decode test at 33.9 tok/s end-to-end. Interactive UI observations reached 40+ tok/s instantaneous/rolling peak. Neither number is a cross-machine guarantee: decode speed depends strongly on prompt and completion length, MTP acceptance, memory pressure, MTPLX version and measurement method. Report the completion length, prompt length and whether a number is decode-only, end-to-end average or rolling peak when comparing results.

The recommended starting profile is sustained with depth 3. Test depth 2 and depth 3 against your own workload; MTP can lose its advantage on some prompts when draft acceptance falls.

Known limitations

  • The MTP sidecar is BF16 by design, so disk/memory characteristics differ from a fully generic 4-bit MLX conversion.
  • MTP is not guaranteed to accelerate every prompt; low acceptance can remove the benefit.
  • max-tokens limits one response, not model context capacity.
  • A long idle period followed by a first request includes model loading and warm-up time.

Citation

Please cite the upstream Qwen3.8 model and link this derivative repository when redistributing or benchmarking it.