Qwen3.8-27B-Uncensored — MLX 5.0 bpw (gs64, NAX-first)
A mixed-precision MLX affine quantization of orcarouter/Qwen3.8-27B-Uncensored
(27.8 B dense, 64 layers, hybrid GatedDeltaNet + full attention, 256 K context, MTP head, vision tower),
built to replicate the recipe of True2456/Qwen3.8-27B-AWQ-5.0bpw
and tuned for oMLX on M5 (NAX tensor unit).
17.38 GB · 5.006 bpw · 3.2× smaller than bf16
| Quantized | 589 modules, 27.640 B params, 17.10 GB |
| Kept bf16 | 610 tensors, 0.283 GB (norms, conv1d, A_log, dt_bias, pos_embed, the 27 vision linear_fc2 whose K=4304 has no legal group size) |
| Format | affine mode: affine — packed U32 weights, BF16 scales and biases |
| MTP head | 31 tensors retained, mtp_num_hidden_layers: 1 |
| NAX | 100 % of quantized bytes in MLX's NAX grid; 89.1 % also clears oMLX's prefill routing gate |
Credits
- True2456 — the recipe this build replicates: the per-module bit
map and its rationale, the group-size-64 argument for oMLX's NAX prefill path, the no-3-bit constraint
(oMLX's
qwen35_prefillkernels ship q4/q5/q6/q8 only), and the MTP and norm-convention findings. Built on their mlx-compress. - OrcaRouter — the bf16 source weights,
Qwen3.8-27B-Uncensored(abliterated fine-tune, revision11a200330368fd04fa7b2f0b1e7f20bd819cf621). - Qwen — the original
Qwen/Qwen3.8-27B.
Per-module widths
| Module | bits | group size |
|---|---|---|
MLP gate_proj / up_proj / down_proj |
4 | 64 |
GDN in_proj_{qkv,z,a,b} |
5 | 64 |
GDN out_proj |
4 | 64 |
attention q/k/v_proj |
8 | 64 |
attention o_proj |
4 | 64 |
embed_tokens |
4 | 128 |
lm_head |
6 | 128 |
vision qkv / proj |
8 | 64 |
vision linear_fc1, merger |
8 | 128 |
MTP attention / fc / MLP |
8 / 6 / 4 | 64 |
No 3-bit anywhere, and gs64 on everything that can route — gs128 is refused by oMLX's NAX path when the tensor unit is available.
How this differs from the recipe it replicates
- SLQ-style GPTQ on the 8-bit groups instead of plain RTN: one-shot block GPTQ, asymmetric,
rel_damp 1e-2, Hessians from real prefill activations of this checkpoint. Measured on the stored bytes: 48/48 attention modules improved, median −32 % H-weighted output error, +31.2 % aggregate. It re-rounds on the grid the export already chose, so only packed codes change. - Vision
qkv/projat gs64 rather than gs128 (+4 MB), because oMLX refuses gs128 on the NAX path — gs64 is what actually reaches the kernel. - No AWQ on the MLP. It was implemented and measured: +32.8 % in-sample, but −36.6 % on held-out prompts, i.e. the per-channel scales and per-group clip ranges fitted the calibration activations rather than the model. Reverted rather than shipped. The MLP carries the uniform-grid range search instead.
Running it
Serve through oMLX. Stock mlx_vlm.load fails on the 31 MTP tensors (Received 31 parameters not in model) and its should_shift_norm_weights heuristic would double-shift norms that are already in
MLX convention here. oMLX gates that heuristic on conv1d layout instead. For a trunk-only load,
pass strict=False.
- Enable native MTP: the head ships inside the checkpoint; drafts are rejection-verified by the trunk, so head error costs acceptance rate, never output correctness.
- The chat template takes
reasoning_effortofxhigh(default),mediumorlowand raises on anything else, plusenable_thinkingandpreserve_thinking— pass viachat_template_kwargs. - For speculative prefill, use a drafter from the same tokenizer family (248,320 vocab), e.g.
Qwen/Qwen3.5-0.8BorQwen/Qwen3.5-2B; a smaller-vocab drafter scores out-of-range ids as zeros.
Measured quality
Per-tensor, dequantized stored bytes vs the bf16 source (rel-L2 / cosine):
| family | bits/gs | rel-L2 | cosine |
|---|---|---|---|
| attention q/k/v (48 modules, GPTQ) | 8/64 | 0.00654 | 0.999978 |
| vision qkv/proj | 8/64 | 0.00665 | 0.999978 |
| vision fc1/merger | 8/128 | 0.00714 | 0.999974 |
| MTP attention | 8/64 | 0.00583 | 0.999983 |
| lm_head | 6/128 | 0.02392 | 0.999714 |
| GDN in_proj | 5/64 | 0.04090 | 0.999163 |
| MLP | 4/64 | 0.08260 | 0.996583 |
| attention o_proj | 4/64 | 0.08308 | 0.996543 |
| GDN out_proj | 4/64 | 0.08445 | 0.996428 |
| embed_tokens | 4/128 | 0.09204 | 0.995755 |
Vision 8-bit and the MTP attention keep the range search rather than GPTQ: no image calibration data was available, so their Hessians could not be captured honestly.
License
Apache-2.0, inherited from the source checkpoint and the Qwen base model. The quantization changes
weights only; behaviour and any limitations of Qwen3.8-27B-Uncensored carry over unchanged.