Qwen3.8-Flash-Next for mlx-serve, 8-bit

mlx-serve pack of Qwen/Qwen3.8-Flash-Next, the Qwen4 preview architecture (model_type: qwen4_exp). Every quantized tensor is 8-bit group 64, the closest-to-bf16 pack in the family. Includes the MTP head and the vision tower (image and video input).

Only use this pack if your Mac has more than 128 GB. It is 137 GB resident.

Download includes the n-gram table, which is mmapped and not resident (see below).

Download MLXServe.com

mlx-serve --model ddalcu/Qwen3.8-Flash-Next-MLX-Serve-8bit --serve

What is different about this model

This is not a Qwen3.5-style pack. Three things around the usual GDN + MoE trunk:

• Gated residual streams. The residual is 4 streams wide (4 x 2560). Every block reads a sigmoid-mixed average of the normalized streams and writes back through per-stream scalar gates. The final mixer replaces the usual final norm.
• N-gram embedding (51B parameters). A second embedding table indexed by hashed bigrams and trigrams of the token ids: 16 heads, each a prime-sized bucket space of ~20M rows, 160 dims per row, injected once before layer 1. It is a lookup, no compute, which is why Qwen quotes the model as 125B: the full checkpoint is 125B trunk + 51B n-gram + 4B MTP = 180B (360 GB bf16).
• Qwen Sparse Attention. Past 2048 tokens each attention layer only reads the 512 most relevant 4-token blocks per query (picked by a small indexer), plus the query's own partial block. Attention cost stays flat with context. Native 262k context.

How this pack stores the n-gram table

The 51B table is NOT in the safetensors shards. It is one merged 8-bit table in ngram_table.bin (57.6 GB, safetensors format, .bin so nothing mlx-loads it). mlx-serve mmaps the file and, per token, dequantizes the 16 rows it needs on the CPU (16 x 160 bytes) and uploads only the resulting 2560-vector. The table never becomes resident: its cost is page cache, which the OS evicts as needed. That is the difference between this pack and mlx-lm style packs that ship the table as 128 quantized tensors and load it onto the GPU (+58 GB resident at this width).

Expected effect: decode speed unchanged (16 tiny reads against a ~20 ms step), cold-cache prefill of very long prompts may pay up to ~1 s per 8k tokens of random reads on the SSD, warm cache is free. No user-space cache is needed, the page cache already is an LRU over exactly this access pattern.

Widths

Every (1 + w) RMSNorm has the +1 folded into the stored weight; depthwise convs are transposed to MLX's [C, K, 1]; experts.gate_up_proj is split into switch_mlp.gate_proj / up_proj. The vision tower ships dense bf16 in model-vision.safetensors (~0.9 GB).

Serving notes

• Memory. ~137 GB resident plus KV cache. mlx-serve sizes the context to what fits; --kv-quant 8 halves the cache.
• MTP. The checkpoint's own 1-layer speculative head is loaded from the pack and drafts by default (--no-mtp or per-request "enable_mtp": false turns it off).
• Concurrency. Text requests batch-decode together; the prefix cache is on, images included, so follow-up turns skip the re-prefill.
• Thinking is on by default ("enable_thinking": false turns it off). Tools use Qwen3.8's XML call format; mlx-serve parses and schema-coerces it.
• Images and video go through the Qwen3-VL-style tower (model.visual.*, dense bf16). MTP is declined on image turns (serial decode).

Conversion

tests/convert_qwen38_flash_next.py in the mlx-serve repo. It streams the 360 GB bf16 checkpoint shard by shard from the Hub (download, quantize, delete), so it converts on a machine with ~150 GB free. The engine was validated against HF transformers (trunk) and the vLLM/SGLang MTP math on a tiny random model before the full conversion.