Mixed-precision MLX build of Motif-Technologies/Motif-3 (314.8B total / 13.2B active MoE, MIT) for Apple silicon — the speed tier companion to avlp12/Motif-3-Alis-MLX-8bit. Decode is read-bandwidth-bound at batch 1, so halving the bits nearly halves the per-token read: ~50 tok/s with MTP self-speculative decoding (k=3) on a single M3 Ultra — about +35% over the 8-bit build at matching settings — at a small, measured quality cost (tables below).
| Build | Size | bpw | Decode (M3 Ultra, MTP k=1) |
|---|---|---|---|
| 8-bit (reference) | 312 GiB | 8.50 | 37.1 tok/s |
| this repo | 167 GiB | 4.55 | 50.2 greedy (peak 53) |
Fits a 256 GB Mac with full headroom; a 192 GB M2/M3 Ultra fits with a raised wired limit and modest context.
| Metric | Value |
|---|---|
| Decode, plain | 28.4 tok/s |
| Decode, MTP k=3 greedy | 50.2 tok/s (5-prompt avg, range 43–53) |
| Decode, MTP k=3 @ T=0.8 | 32.1 tok/s (equality acceptance; rejection sampling is a k=1-only option — sampled drafts degrade chained anchors) |
| Prefill | ~790 tok/s @ 8k on natural mixed text · ~580 @ 2k (chunked, step 2048; ≈75% of the measured bf16-GEMM ceiling). Highly repetitive prompts can drop to ~545: expert-routing skew fattens the small-group tail in gather_qmm (upstream report) |
| Peak memory | ~180 GB |
Speed stack (all in the install fork):
single-kernel Sinkhorn for the mHC gates, compiled activation/glue fusions (PolyNorm,
router, mHC mixes, attention epilogue — bit-exact, kill switches MOTIF_COMPILE_ACT/GLUE=0),
an mHC-transition mega-kernel (one threadgroup per position fuses the 16K rmsnorm, gate
projections, Sinkhorn, premix, and input layernorm — ~7 serial dispatches to 1, +17%;
MOTIF_MHC_TRANS=0),
and vendor-faithful MTP self-speculative decoding (wiring confirmed by Motif in
this discussion) with
correct chained-draft norm handling. k=3 drafting wins on Apple silicon: batch-1 decode
is latency-bound, so verifying 4 tokens costs only ~1.5x one token — the economics invert
Motif's (compute-bound, vLLM) "1 speculative token is optimal" guidance. Leviathan
rejection-sampling acceptance remains available for k=1 (MLX_MTP_REJECTION=0 to disable).
Teacher-forced NLL and greedy top-1 agreement against the 8-bit reference on short KO/EN/code probes (same final-release weights; probe-scale — treat as indicative, not benchmark-grade. *top-1 measured on the shorter 300-token probe, on which the KO NLL delta read +9.9% — probe-scale variance; the 1.5k-token probe is the better estimate):
| Probe | NLL Δ vs 8-bit | top-1 agreement |
|---|---|---|
| Korean prose (1.5k-token probe) | +2.4% | 89.0%* |
| English prose | −0.8% | 89.1% |
| Python code | −3.2% | 98.8% |
If you want maximum fidelity (e.g. as a distillation teacher), use the 8-bit build. Motif themselves ship a 4-bit NVFP4 checkpoint for vLLM, and report the MTP head keeps ≥70% draft acceptance at 4-bit — consistent with what this build measures.
C6 mixed profile: experts 4-bit g64, attention/dense/shared-expert 6-bit, wkv_a 8-bit
(latent chokepoint), router gate / mHC / lambda_proj / norms / PolyNorm coefficients in
bf16. MTP head 8-bit (model-mtp.safetensors, +254 MB). Per-tensor map in config.json.
Tensors >2³¹ elements sanitized without mx.split (mlx#3836 workaround, byte-verified).
pip install git+https://github.com/avlp12/mlx-lm.git@motif3-support
mlx_lm.generate --model avlp12/Motif-3-Alis-MLX-4.5bpw \
--prompt "한국의 전통 건축양식을 요약해줘." --max-tokens 400 --temp 0.7 --mtp --mtp-num-draft-tokens 3
# OpenAI-compatible server:
mlx_lm.server --model avlp12/Motif-3-Alis-MLX-4.5bpw --port 8003 --mtp --mtp-num-draft-tokens 3
--mtp-num-draft-tokens 3 is optimal on Apple silicon (latency-bound verify makes
wide drafting cheap; k=2 is within noise, k=1 is ~−20%). Motif's own "1 speculative
token" guidance applies to compute-bound GPU serving, not this regime.model-00037-of-00039.safetensors 4.7 GBmodel-00025-of-00039.safetensors 4.7 GBmodel-00022-of-00039.safetensors 4.7 GBmodel-00013-of-00039.safetensors 4.7 GBmodel-00010-of-00039.safetensors 4.7 GBmodel-00031-of-00039.safetensors 4.7 GBmodel-00034-of-00039.safetensors 4.7 GBmodel-00016-of-00039.safetensors 4.7 GBmodel-00019-of-00039.safetensors 4.7 GBmodel-00028-of-00039.safetensors 4.7 GBmodel-00007-of-00039.safetensors 4.7 GBmodel-00004-of-00039.safetensors 4.7 GBmodel-00001-of-00039.safetensors 4.6 GBmodel-00014-of-00039.safetensors 4.6 GBmodel-00038-of-00039.safetensors 4.6 GBmodel-00012-of-00039.safetensors 4.6 GBmodel-00023-of-00039.safetensors 4.6 GBmodel-00009-of-00039.safetensors 4.6 GBmodel-00015-of-00039.safetensors 4.6 GBmodel-00027-of-00039.safetensors 4.6 GBmodel-00030-of-00039.safetensors 4.6 GBmodel-00036-of-00039.safetensors 4.6 GBmodel-00018-of-00039.safetensors 4.6 GBmodel-00024-of-00039.safetensors 4.6 GBmodel-00026-of-00039.safetensors 4.6 GBmodel-00020-of-00039.safetensors 4.6 GBmodel-00021-of-00039.safetensors 4.6 GBmodel-00017-of-00039.safetensors 4.6 GBmodel-00005-of-00039.safetensors 4.6 GBmodel-00003-of-00039.safetensors 4.6 GBmodel-00006-of-00039.safetensors 4.6 GBmodel-00033-of-00039.safetensors 4.6 GBmodel-00008-of-00039.safetensors 4.6 GBmodel-00011-of-00039.safetensors 4.6 GBmodel-00029-of-00039.safetensors 4.6 GBmodel-00032-of-00039.safetensors 4.6 GBmodel-00002-of-00039.safetensors 4.6 GBmodel-00035-of-00039.safetensors 4.6 GBmodel-00039-of-00039.safetensors 3.0 GBmodel-mtp.safetensors 267 MB有文件?提交种子。
提交种子