Mixed-precision affine quantization of
orcarouter/Qwen3.8-Flash-Next-Uncensored
(qwen4_exp, 180B) sized for an M5 Max 128 GB at up to 262k context.
134.30 GB · 5.969 bpw · 180.0B params · 2.68× vs bf16 (360 GB)
This is v2.2, the revision after v2: same family map, but every per-group quantization range is chosen against the importance-weighted objective (oQe), the n-gram table moves to 8-bit, the whole artifact sits on a single 4-or-8-bit group-size-64 lattice — and therefore on the runtime's INT8-prefill (A8) grid. It is +25.01 GB (+22.9 %) over v2 and −0.2 GB resident, and it leads v2 on the output-gated comparison: 72.3 vs 70.8 overall, 16/20 vs 15/20 at the 96-token budget.
(scale, bias) — and the search that picks it — is driven by an importance matrix
accumulated over 65,536 calibration tokens (128 sequences × 512) of the dsh_verify RAG
corpus, then refit by weighted least squares. On identical tensors this trades plain error
for weighted error: experts measure 0.09130 plain / 0.07662 weighted against v2's 0.08538
plain full-sweep figure, i.e. 10.3 % lower on the objective the quantizer optimises,
6.9 % higher on the one it does not. The L1 gate is therefore read on weighted.mmap, so the extra
25.6 GB costs 0 resident memory. 8/32 is not a preference but the ceiling — the runtime
ships a Metal dequantize kernel only for gs 32 at 8 bits on 160-wide rows; gs 8/16/20/40/80/160
have no kernel at all.switch_mlp.{gate,up,down}_proj rather than v2's fused source layout — 294 fused tensors
repacked into 441 stacked ones, byte-identical payload — so the artifact loads through the
current qwen4_exp path (and through mlx-vlm's own split_expert_projection) with no
runtime patch.131 safetensors shards (model-00001…00131-of-00131) plus the MTP sidecar
(model-mtp.safetensors), config.json, generation_config.json, chat_template.jinja,
tokenizer.json / tokenizer_config.json / vocab.json / merges.txt,
preprocessor_config.json, video_preprocessor_config.json. 3,781 tensors:
133.77 GB quantized + 0.54 GB exact bf16. The vision tower is inline in the main shards; the
MTP head is the sidecar. Routed experts are stored stacked (switch_mlp.*).
The reference is the bf16 weights of the same source checkpoint for tensor-level error, and the same hosted bf16 oracle + fixed 21-case corpus that scored v2 for output-level numbers, so the v2 column is directly comparable.
| metric | v2 | v2.2 (this model) |
|---|---|---|
| Artifact | 109.29 GB · 4.857 bpw | 134.30 GB · 5.969 bpw |
| Quantized / exact modules | 743 / 761 | 1037 / 466 |
Resident (PLE mmap) |
76.9 GB | 76.70 GB |
| N-gram table rel-L2 (per shard) | 0.07335 (4/32) | 0.00484 (8/32, −93.4 %) |
| Routed experts rel-L2 | 0.08538 (plain, full sweep) | 0.09130 plain / 0.07662 weighted |
Dense rel-L2 (lm_head, 8/64) |
— | 0.00600 |
| L2 oracle, 96-token budget | 70.77 / 100, 15/20 | 72.27 / 100, 16/20 |
| Top-1 agreement with the bf16 oracle | 90.7 % | 86.2 % |
| A/B against v2, same captures | — | 16/21 identical, prefix agreement 0.9902, mean top-1 0.8868, mean JSD 0.05355 |
| INT8 prefill (4,096 tok) | not on the A8 grid | 1.08× (987 → 1,066 tok/s) |
Metric discipline. The experts row is not a like-for-like plain-error comparison: v2's 0.08538 is a full 1,707-tensor sweep; v2.2's 0.09130/0.07662 come from the installed artifact on a 2M-element slice. The gate v2.2 is accepted on is the importance-weighted one (0.07662 vs the 0.085 gate), which is the objective oQe optimises; its plain figure is above the older plain gate by design. The 96-token row is the like-for-like pair — both captures were scored against the same oracle, same corpus, same budget. Top-1 agreement with the oracle is lower than v2's even though the task score is higher: agreement is baseline-relative, so a build that rephrases correctly scores lower on it.
Base format: MLX affine — packed U32 weights + bf16 scales + bf16 biases, i.e.
2 × 16 / group_size bits of overhead per weight on top of the nominal width. 1,121 explicit
per-module config entries; 1,037 quantized modules.
| Module | Bits | Group | Params | Size | Rationale |
|---|---|---|---|---|---|
| Routed experts MLP (48 × 512) | 4 | 64 | 120.8B | 67.95 GB | dominant weight mass; the only grid with an expert-A8 prefill path and the fused MoE decode kernel |
| N-gram PLE table (128 shards) | 8 | 32 | 51.2B | 57.60 GB | mmap → 0 resident; gs32 is structural (rows are 160 wide) and the only loadable 8-bit grouping |
Dense GDN + attention (in_proj_*, out_proj, q/k/v/o_proj) |
8 | 64 | 2.70B | 2.89 GB | W8A8 prefill path, A8-eligible |
| MTP draft head (experts, attn, shared, FC, indexer) | 8 | 64 | 2.61B | 2.78 GB | the whole head on one lattice; the trunk verifies every token, so its precision costs acceptance |
lm_head, embed_tokens |
8 | 64 | 1.27B | 1.35 GB | A8-eligible; lm_head is the largest single matvec per step |
| Shared experts | 8 | 64 | 0.24B | 0.25 GB | A8 grid (v2's 8/128 was off it); costs +7.5 MB |
| Hyper-connections (per-layer matmuls) | 8 | 64 | 0.64B | 0.69 GB | quantized — the fused HC kernels require it; v2 kept these bf16 |
| Vision tower | 8 | 64 | 0.45B | 0.60 GB | one lattice with everything else; linear_fc2 and patch_embed admit no group size at any width and stay exact |
| Everything else (norms, routers, convs, trunk indexer, PLE machinery) | exact bf16 | — | 0.10B | 0.19 GB | routing is discrete, norms are cheap |
Byte accounting measured from the checkpoint: 133.77 GB quantized + 0.54 GB exact bf16.
100 % of quantized bytes are eligible — 133.77 of 133.77 GB, verified against the kernel grid
that ships inside the app's mlx.metallib rather than from documentation, with the artifact's
own module list:
| path | format | modules | GB | NAX kernel |
|---|---|---|---|---|
| gather | affine 4/64 (trunk experts) | 144 | 67.95 | yes |
| dense | affine 8/32 (n-gram PLE shards) | 128 | 57.60 | yes |
| dense | affine 8/64 (attn/GDN/lm_head/vision/HC/MTP-dense) | 762 | 5.55 | yes |
| gather | affine 8/64 (MTP experts) | 3 | 2.67 | yes |
Two caveats, unchanged from v2. Eligibility is not engagement — the dispatch heuristic lives
in compiled host code. Measured: the quantized dense NAX kernels reach 31.7 / 34.6 / 33.3 TFLOP/s
at gs32-b8 / gs64-b4 / gs64-b8 against a 50.9 TFLOP/s bf16 reference, i.e. uniformly
1.47–1.61× slower — the expected small-tile penalty at compute-bound sizes, with no SIMD
fallback outlier. The gather path is engaged in the served runtime (sorted gather_qmm reroute
installed, MoE routed-expert decode engaged, routed A8 gate/up on 48 layers, 35–57 tok/s at
14k–25k-token prompts with MTP drafting). And at prefill sizes quantization is still the slower
option: the A8 path is what recovers part of that (1.08× over the same artifact's own A8-off
run), not a bf16 parity claim — a bf16 prefill baseline does not fit on this machine.
Streaming, tensor-by-tensor quantization; the map is applied from a header-only preflight
(export_v5.py --validate-map) that reads every source tensor and asserts the result before any
bytes are written. 1658 source tensors, 0 unmatched.
dsh_verify RAG database: 128 sequences × 512
tokens, expert coverage capped at 1,024 sequences (~6–9 min of collection on this machine).
It measures per-input-channel importance per module, which is what the weighted refit and the
oQe range search consume.(scale, bias) pairs are scored against the importance-weighted
error of mx.dequantize output on the stored bf16 parameters — the runtime's own
dequantizer — and the built-in min/max result is always among the candidates, so no group can
come out worse than v2's rule on the objective being scored.switch_mlp tensors (weights, scales and biases moved together; byte-identical payload,
verified by mx.array_equal on a gate/up pair and a down projection) and the config entries
were re-keyed. This is what makes the artifact loadable on the current loader without patches.qwen4_exp recipient, so the merge is
explicit.Needs oMLX 0.8.0rc1 (the release that resolves its own vendored qwen4_exp) and an M5-class
GPU. Verified on an M5 Max 128 GB: loads in 11.0 s, 75.01 GB resident, generates through
the OpenAI API, MTP drafting at 3.2 tok/cycle with 9/10 accepted on a smoke test, and
92.3 tok/s with 0.12 s TTFT on the recorded session.
mode="mmap" (qwen4_ple_ssd_offload). Residency with the 57.6 GB table in RAM
does not fit; the engine forces the table to SSD and serves it through page cache.a8_prefill_enabled: true, a8_prefill_min_tokens: 128. These are model settings, not
request parameters. Below the floor, prefill stays on the bf16 path.mtp_enabled: true. Lightning MTP decode; the "enable_mtp" request field is ignored.sudo sysctl iogpu.wired_limit_mb=120000.| rung | tool | result |
|---|---|---|
| L0 structure | audit_mlx_quant.py --profile v5 |
all structural checks passed — 3,781 tensors, 1,037 quantized + 466 exact modules, complete weight/scales/biases triplets, index ↔ disk parity |
| L1 weights | l1_v5.py |
dense 0.00600, PLE 0.00484, experts 0.09130 plain / 0.07662 weighted — all within their gates, experts on the weighted gate |
| L2 outputs | compare_captures.py, oracle.py |
A/B vs v2: 16/21 identical, prefix 0.9902, top-1 0.8868; oracle 72.27/100, 16/20 |
| A8 prefill | bench_a8_v5.py |
4,096-token prefill 987 → 1,066 tok/s (1.08×) with INT8 on |
| NAX | nax_compat.py |
100 % of quantized bytes eligible; dense engagement measured, gather path engaged in the serving logs |
| Serving | oMLX API | loads in 11.0 s (PLE mmap, no Hadamard patch), generates, MTP active |
-Uncensored variant changed relative to Qwen's
official model. The v2 ↔ v2.2 comparison is sound because both were measured the same way
against the same reference; the absolute scores are not a quantization-error measurement.Quantization recipe and measurement tooling by the qwen3.8-next-flash project. Base weights:
orcarouter/Qwen3.8-Flash-Next-Uncensored. Architecture and the original model: Qwen.
Quantizer/runtime: oMLX and MLX. Licensed Apache-2.0,
following the base model.
model-mtp.safetensors 2.8 GBmodel-00035-of-00131.safetensors 1.8 GBmodel-00032-of-00131.safetensors 1.8 GBmodel-00036-of-00131.safetensors 1.8 GBmodel-00018-of-00131.safetensors 1.8 GBmodel-00031-of-00131.safetensors 1.8 GBmodel-00033-of-00131.safetensors 1.8 GBmodel-00028-of-00131.safetensors 1.8 GBmodel-00034-of-00131.safetensors 1.8 GBmodel-00009-of-00131.safetensors 1.8 GBmodel-00030-of-00131.safetensors 1.8 GBmodel-00012-of-00131.safetensors 1.8 GBmodel-00016-of-00131.safetensors 1.8 GBmodel-00017-of-00131.safetensors 1.8 GBmodel-00026-of-00131.safetensors 1.8 GBmodel-00008-of-00131.safetensors 1.8 GBmodel-00011-of-00131.safetensors 1.8 GBmodel-00015-of-00131.safetensors 1.8 GBmodel-00019-of-00131.safetensors 1.8 GBmodel-00021-of-00131.safetensors 1.8 GBmodel-00022-of-00131.safetensors 1.8 GBmodel-00025-of-00131.safetensors 1.8 GBmodel-00013-of-00131.safetensors 1.8 GBmodel-00020-of-00131.safetensors 1.8 GBmodel-00027-of-00131.safetensors 1.8 GBmodel-00029-of-00131.safetensors 1.8 GBmodel-00024-of-00131.safetensors 1.8 GBmodel-00010-of-00131.safetensors 1.8 GBmodel-00014-of-00131.safetensors 1.8 GBmodel-00023-of-00131.safetensors 1.8 GBmodel-00006-of-00131.safetensors 1.8 GBmodel-00007-of-00131.safetensors 1.8 GBmodel-00005-of-00131.safetensors 1.4 GBmodel-00130-of-00131.safetensors 1.2 GBmodel-00037-of-00131.safetensors 982 MBmodel-00038-of-00131.safetensors 944 MBmodel-00040-of-00131.safetensors 944 MBmodel-00042-of-00131.safetensors 944 MBmodel-00044-of-00131.safetensors 944 MBmodel-00046-of-00131.safetensors 944 MBmodel-00048-of-00131.safetensors 944 MBmodel-00050-of-00131.safetensors 944 MBmodel-00052-of-00131.safetensors 944 MBmodel-00056-of-00131.safetensors 944 MBmodel-00061-of-00131.safetensors 944 MBmodel-00063-of-00131.safetensors 944 MBmodel-00065-of-00131.safetensors 944 MBmodel-00067-of-00131.safetensors 944 MBmodel-00069-of-00131.safetensors 944 MBmodel-00071-of-00131.safetensors 944 MBmodel-00073-of-00131.safetensors 944 MBmodel-00075-of-00131.safetensors 944 MBmodel-00077-of-00131.safetensors 944 MBmodel-00083-of-00131.safetensors 944 MBmodel-00085-of-00131.safetensors 944 MBmodel-00087-of-00131.safetensors 944 MBmodel-00089-of-00131.safetensors 944 MBmodel-00091-of-00131.safetensors 944 MBmodel-00093-of-00131.safetensors 944 MBmodel-00095-of-00131.safetensors 944 MBmodel-00097-of-00131.safetensors 944 MBmodel-00099-of-00131.safetensors 944 MBmodel-00101-of-00131.safetensors 944 MBmodel-00105-of-00131.safetensors 944 MBmodel-00107-of-00131.safetensors 944 MBmodel-00109-of-00131.safetensors 944 MBmodel-00111-of-00131.safetensors 944 MBmodel-00115-of-00131.safetensors 944 MBmodel-00119-of-00131.safetensors 944 MBmodel-00054-of-00131.safetensors 944 MBmodel-00079-of-00131.safetensors 944 MBmodel-00117-of-00131.safetensors 944 MBmodel-00113-of-00131.safetensors 944 MBmodel-00002-of-00131.safetensors 944 MBmodel-00004-of-00131.safetensors 944 MBmodel-00058-of-00131.safetensors 944 MBmodel-00081-of-00131.safetensors 944 MBmodel-00103-of-00131.safetensors 944 MBmodel-00121-of-00131.safetensors 944 MBmodel-00123-of-00131.safetensors 944 MBmodel-00125-of-00131.safetensors 944 MBmodel-00129-of-00131.safetensors 944 MBmodel-00127-of-00131.safetensors 944 MBmodel-00001-of-00131.safetensors 686 MBmodel-00131-of-00131.safetensors 675 MBmodel-00076-of-00131.safetensors 612 MBmodel-00086-of-00131.safetensors 612 MBmodel-00112-of-00131.safetensors 612 MBmodel-00057-of-00131.safetensors 612 MBmodel-00120-of-00131.safetensors 612 MBmodel-00049-of-00131.safetensors 612 MBmodel-00126-of-00131.safetensors 612 MBmodel-00102-of-00131.safetensors 612 MBmodel-00041-of-00131.safetensors 612 MBmodel-00082-of-00131.safetensors 612 MBmodel-00094-of-00131.safetensors 612 MBmodel-00068-of-00131.safetensors 612 MBmodel-00070-of-00131.safetensors 555 MBmodel-00043-of-00131.safetensors 555 MBmodel-00064-of-00131.safetensors 555 MBmodel-00062-of-00131.safetensors 555 MBmodel-00078-of-00131.safetensors 555 MBmodel-00108-of-00131.safetensors 555 MBmodel-00104-of-00131.safetensors 555 MBmodel-00053-of-00131.safetensors 555 MBmodel-00114-of-00131.safetensors 555 MBmodel-00116-of-00131.safetensors 555 MBmodel-00090-of-00131.safetensors 555 MBmodel-00096-of-00131.safetensors 555 MBmodel-00106-of-00131.safetensors 555 MBmodel-00122-of-00131.safetensors 555 MBmodel-00098-of-00131.safetensors 555 MBmodel-00088-of-00131.safetensors 555 MBmodel-00051-of-00131.safetensors 555 MBmodel-00045-of-00131.safetensors 555 MBmodel-00128-of-00131.safetensors 555 MBmodel-00072-of-00131.safetensors 555 MBmodel-00003-of-00131.safetensors 555 MBmodel-00059-of-00131.safetensors 494 MBmodel-00055-of-00131.safetensors 494 MBmodel-00092-of-00131.safetensors 494 MBmodel-00047-of-00131.safetensors 494 MBmodel-00084-of-00131.safetensors 494 MBmodel-00066-of-00131.safetensors 494 MBmodel-00080-of-00131.safetensors 494 MBmodel-00118-of-00131.safetensors 494 MBmodel-00074-of-00131.safetensors 494 MBmodel-00039-of-00131.safetensors 494 MBmodel-00100-of-00131.safetensors 494 MBmodel-00124-of-00131.safetensors 494 MBmodel-00110-of-00131.safetensors 494 MBmodel-00060-of-00131.safetensors 62 MBorcarouter/Qwen3.8-Flash-Next-Uncensored