Qwen3.8-2.4T-A95B-MLX-reap75-4bit

MLX build of Qwen3.8-2.4T-A95B — 2.4 T parameters, 95 B active, 92 layers, 512 experts routed top-10 — quantized to 4-bit (group size 64) with REAP pruning to 25% of experts (128 of 512).

These files are modified: converted to MLX and quantized, and experts pruned. The architecture is otherwise unchanged.

Size

Experts are 99.98% of this model's weights, so the bit width essentially is the file size.

You need the runtime

qwen3_5_moe_text is not carried by mlx-lm. Use the port:

git clone https://github.com/PipeNetwork/qwen38-mlx
cd qwen38-mlx && pip install -r requirements.txt

from qwen38_mlx.load import load
model, tokenizer = load("/path/to/Qwen3.8-2.4T-A95B-MLX-reap75-4bit")

It is validated against transformers to 1.9e-06 on hidden states, with a negative control for each fragile behaviour (layer-type dispatch, partial rotary, top-k routing) showing that breaking it changes the output.

Quality

Perplexity is not reported here, and the reason matters. The full-size builds cannot be run — at 550 GB of memory, even 2-bit is 757 GB. Quoting a number obtained some other way, or none at all, would both be worse than measuring what can actually be measured.

A single decoder layer is 51.6 GB and does fit. So every layer was run twice on identical inputs — once in bf16, once quantized — and the divergence recorded, in two modes:

• teacher-forced: each build sees the bf16 hidden states, isolating that layer's own damage.
• free-running: each build feeds itself, which is what inference does, and where damage compounds.

Both are paired by construction: same layer, same 16,384 tokens, differing only in the weights.

Read the last two columns together. Teacher-forced error understates the damage by up to 46x — 8-bit is 0.0018 per layer but 0.083 once compounded over 92 of them. Cosine says whether the activation still points the same way, which is what survives best under quantization.

What the ladder shows

1. REAP-50 + 3-bit beats full + 2-bit on both axes at once — smaller (541 GB vs 757 GB) and less damaged (0.487 vs 0.704 free-running). If you are memory-bound, dropping half the experts and spending the savings on precision is the better trade.
2. Past REAP-50, bit width stops mattering. REAP-75 at 6-bit (0.723) and at 4-bit (0.729) are indistinguishable — expert removal dominates completely, so the extra 160 GB buys nothing.
3. 2-bit and REAP-75 are both cliffs. Every build that avoids both is materially better than every build that hits either.

Speed: a throughput model, not an interactive one

Measured on an M3 Ultra / 550 GB with the 361 GB build:

A forward pass costs a fixed ~4 seconds almost regardless of how many tokens are in it — 3.86 s for one token, 5.45 s for 256, 6.59 s for a batch of 128. The 92 decoder layers account for only 0.18 s of that; the remainder is per-dispatch overhead over 3,572 large resident buffers.

Three explanations were tested and ruled out, so save yourself the trouble:

• Not the GPU wired-memory limit. Raising iogpu.wired_limit_mb from its ~412 GB default to 500 GB changed nothing.
• Not paging or memory mapping. Copying every weight out of its mmap into MLX-owned memory changed nothing; the cost is identical warm and cold.
• Not model size, except weakly. 437 GB cost 7.1 s per forward and 361 GB cost 4.6 s — it scales far less than proportionally, which is what fixed overhead looks like.

The practical consequence: batch your requests. One token at a time is 0.2 tok/s and unusable interactively; 128 sequences at a time is ~19 tok/s aggregate off the same 4-second step. If you want an interactive model on a single Mac, this is the wrong one at any quantization.

REAP

Experts were ranked by REAP saliency — router_weight x ||expert_output|| — accumulated over 65,536 calibration tokens, one layer at a time so the 4.8 TB model never had to be resident.

Two things are worth stating plainly about that ranking, since pruning is permanent:

• 20 of 47104 experts were never routed to at all across 13 layers. Those are free to drop.
• The calibration set was split in half and each half ranked independently. Where the halves disagree they swap near-equivalent experts: choosing on one half and scoring against the other retains 98.4% / 95.1% / 90.7% of the oracle's saliency at keep-75/50/25%.

One structural note specific to this architecture: the router takes a softmax over all experts, then top-k, then renormalises the selected weights — so the softmax denominator cancels exactly. Dropping router rows is therefore identical to masking those experts before the top-k, with no renormalisation drift. That is not true of every MoE.

License

Apache-2.0, as upstream. Port: https://github.com/PipeNetwork/qwen38-mlx.