Qwen3.8-Flash-Next-Sushi-2.6bpw
Qwen/Qwen3.8-Flash-Next packed for sushi, a native inference engine for Apple Silicon. Target: a 64 GB Mac.
| part | how it is stored |
|---|---|
| routed experts (48 layers x 512, plus the MTP layer) | EXL3 at 2.6 bits per weight, made with our own converter |
| dense part (attention, GDN, shared experts, hyper-connections, indexer, embedding, lm_head) | 8-bit affine; the same as in Sushi-3bpw and Sushi-4bpw |
| n-gram embedding table | 4-bit affine, group size 32 (mlx-serve's quantization of the released bf16 table), one file ngram_table.bin, read from the SSD and never loaded into GPU memory; the same file as Sushi-3bpw |
| vision tower (Qwen3-VL ViT) | included: image and video input |
| MTP head | included |
| context | up to 1,048,576 tokens (config.json sets YaRN x4 over the native 262,144) |
| disk | ~74 GiB: 43.9 GiB of weights + 29.8 GiB n-gram table |
sushi only. This pack needs sushi v1.0.4 or later on Apple Silicon with macOS 26.2 or later. It does not load in transformers, vLLM, mlx-lm or exllamav3.
How to run it (install, launch flags, memory and context per Mac): sushi usage.
Quality
KLD against the bf16 checkpoint: 16 prompts x 512 tokens, scored to the first EOS (7186 positions), KV cache at kv8 (Sushi-2bpw: bf16 KV), every pack scored by sushi. Lower is better. The n-gram table is not counted: it stays on the SSD.
| pack | weights in GPU memory (GiB) | KLD | top-1 agreement |
|---|---|---|---|
| Sushi-4bpw | 63.68 | 0.0592 | 92.89% |
| oMLX oQ5e | 83.97 | 0.0625 | 92.40% |
| mlx-serve mixed-4-8bit | 70.13 | 0.0818 | 91.39% |
| Sushi-3bpw (4-bit n-gram table) | 49.33 | 0.1036 | 90.31% |
| Sushi-2.6bpw (4-bit n-gram table) | 43.95 | 0.1355 | 89.08% |
| oMLX oQ4e | 69.21 | 0.1370 | 88.87% |
| mlx-serve iQ-MLX 3.3bpw | 50.60 | 0.1987 | 86.28% |
| Vontra 4-bit g32 (TensorFold) | 75.60 | 0.2074 | 85.01% |
| Sushi-2bpw (4-bit n-gram table) | 34.97 | 0.2080 | 85.94% |
Sushi-2.6bpw is the smallest pack here after Sushi-2bpw, the 48 GB pack. It matches oMLX oQ4e's KLD (0.136 vs 0.137) with 25 GiB less, and has about two thirds of mlx-serve's iQ-MLX 3.3bpw KLD (0.136 vs 0.199) with 6.7 GiB less. Sushi-3bpw costs 5.4 GiB more for 24% lower KLD; on a 64 GB Mac that is about half the context. Each pack is measured with the n-gram table it ships: 4-bit g32 in Sushi-2bpw, Sushi-2.6bpw, Sushi-3bpw and mlx-serve's packs, 4- or 5-bit in oMLX's, bf16 in Sushi-4bpw. With the bf16 table, Sushi-2.6bpw reads 0.1303. The oMLX and mlx-serve packs were measured as published; the oMLX n-gram tables were repacked into sushi's file unchanged.
Hard questions
LLMProbe's Hard 50 set (MMLU-Pro, OlympiadBench, LiveBench, NIST Juliet): 44 correct, 4 incorrect, 2 partial (hit the token cap), 88.0%. llmprobe 0.6.12, xhigh effort, 12k reasoning / 16k output tokens, 8-bit KV and MTP.
Choosing the n-gram table
The Sushi packs ship two copies of the same n-gram table, and either copy works with any of them. Use the 4-bit table to save 66 GiB of disk, or the bf16 table for a small quality gain. The table is read from the SSD, so GPU memory is the same either way.
| table | size on disk | ships with | KLD with Sushi-2.6bpw | KLD with Sushi-3bpw | KLD with Sushi-4bpw |
|---|---|---|---|---|---|
| 4-bit, group size 32 | 29.8 GiB | Sushi-2bpw, Sushi-2.6bpw, Sushi-3bpw | 0.1355 | 0.1036 | 0.0654 |
| bf16 | 95.4 GiB | Sushi-4bpw | 0.1303 | 0.1006 | 0.0592 |
To swap, download the other pack's ngram_table.bin into this pack's folder, replacing the current one:
hf download beamster/Qwen3.8-Flash-Next-Sushi-4bpw ngram_table.bin --local-dir <this pack's folder>
Then set ngram_table in config.json to match the file:
"ngram_table": {"file": "ngram_table.bin", "bits": 16, "group_size": 0} for the bf16 table.
Credits and license
- Qwen team: the base model, Qwen/Qwen3.8-Flash-Next. This
pack is a derivative under the Qwen Community License 1.0 (
LICENSE), including its conditions on large commercial services. - turboderp: the EXL3 format.
- ddalcu: mlx-serve. sushi is a fork of it.
- David Tai: his work on --mtp-typical