MiMo-V2.6-Flash-MOPD, fp8res mlx-stream pack
A serving pack of XiaomiMiMo/MiMo-V2.6-Flash-MOPD for mlx-serve with the mlx-stream plugin on Apple silicon. The routed experts (the bulk of the model) stay on disk and stream from the SSD per request; the attention, dense and router weights stay resident. On one M5 Max (128 GB) this serves the 309B-parameter model at about 1,000 tok/s prefill and 30 tok/s decode for a 16K-token prompt, and holds the full 1,048,576-token context.
The weights are the release's: the routed experts are a bit-exact repack of the release's MXFP4 experts into fixed
13,369,344-byte records (experts.bin), and the resident projections keep the release's FP8 (e4m3, 64x128 blocks).
Nothing was re-quantized. License: MIT, inherited from the release; the model is Xiaomi's.
Files (171 GB)
| file | bytes | what |
|---|---|---|
experts.bin |
160,859,947,008 | the expert bank: 47 MoE layers x 256 routed experts, one 13,369,344-byte record each (gate, up, down in the release's MXFP4 group-32 layout), format mlx-stream-mxfp4-expert-bank-v1 |
expert-manifest.json |
2,087,572 | per-record offset, length and sha256; the bank's format and geometry |
model.safetensors |
8,895,389,040 | the residents: embeddings, attention, the dense layer, routers, norms, lm_head; q/k/v and the dense MLP in the release's FP8 |
model.safetensors.index.json |
40,989 | the residents' index |
mtp-heads.safetensors |
1,189,599,424 | the release's multi-token-prediction heads, the draft lane's sidecar |
mtp-manifest.json |
23,723 | the sidecar's binding to this pack (data sha256, payload sizes) |
config.json |
84,703 | the release config plus the expert_streaming block the plugin reads (residents_format: fp8_block128) |
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json |
the release's |
experts.bin must be a real file or a hard link on the serving machine (the stream reads it with uncached I/O).
Serving
Requirements: Apple silicon with 128 GB unified memory (the fp8res pack needs about 104-108 GB at a 16K prompt), the
pack on the internal SSD, mlx-serve built with the mlx-stream plugin on its main (the host from branch pr/mimo-host
until ddalcu/mlx-serve#822 merges), and the wired limit raised:
sudo sysctl iogpu.wired_limit_mb=114688
The command the measurements used (the harness's host/port/timeout flags dropped):
MLX_SERVE_GPU_CEILING_MB=114688 mlx-serve serve --model /abs/path/to/MiMo-V2.6-Flash-MOPD-fp8res-mlx-stream \
--ctx-size 32768 --no-vision --no-warmup-eager --prefix-cache-entries 0 --metrics --log-file off \
--kv-quant off --wired-margin-gib 2
--ctx-size is the context the memory bill covers (prompt plus generation); the plugin fills the resident expert rows
up to the GPU ceiling less the wired margin and logs the fill (MiMo memory: ... prompt_rows= decode_rows=). Every
route is a construction setting in the pack's config.json under expert_streaming.settings; the defaults are the
measured best routes, and the measurements below pinned the rows explicitly, for example:
"expert_streaming": { "settings": { "expert_prefill_rows": 130, "expert_rows": 147, "mtp_lane": true } }
The server speaks the OpenAI-compatible API (/v1/completions, /v1/chat/completions). The plugin's
docs/mimo.md has the full recipe,
every setting, the memory bill and the limitations; docs/mimo-pack.md
describes the pack and how it was built from the release.
Measurements
One M5 Max, 128 GB, one internal SSD (about 13.5 GB/s), 2026-10-11, nothing else loaded, each cell a fresh server and one request. KV bf16, the draft (MTP) lane on, FP8 residents. "Card" = temperature 1.0, top_p 0.95, seed 1, thinking on; the standard 16K prompt is a 16,382-token coding prompt; rows = resident expert rows per layer (prompt / decode).
| cell | rows | prefill tok/s | decode tok/s | TTFT s | answer |
|---|---|---|---|---|---|
| 16K card, 16,384-token answer (mean of 2, identical outputs) | 134 / 152 | 999.8 | 28.25 | 16.4 | 16,384 tokens, cap |
| 16K card, 688-token answer | 130 / 147 | 1,003.9 | 31.6 | 16.3 | complete, stopped on EOS |
| 16K card on the binary of record (same cell, identical output) | 130 / 147 | 998.8 | 32.3 | 16.4 | complete, stopped on EOS |
The card ladder, one cold card request per prompt length, rows pinned per rung to what the memory bill admits:
| prompt tokens | rows | prefill tok/s | decode tok/s | TTFT | answer |
|---|---|---|---|---|---|
| 1,023 | 130 / 147 | 92.3 (cold first request) | 31.6 | 11.1 s | EOS |
| 2,048 | 129 / 146 | 178.3 | 33.1 | 11.5 s | EOS |
| 4,096 | 129 / 146 | 345.5 | 32.2 | 11.9 s | EOS |
| 8,192 | 129 / 146 | 654.4 | 30.9 | 12.5 s | EOS |
| 16,384 | 129 / 146 | 978.5 | 30.7 | 16.7 s | EOS |
| 32,768 | 128 / 144 | 827.8 | 31.3 | 39.6 s | EOS |
| 65,536 | 125 / 142 | 743.9 | 29.1 | 88.1 s | EOS |
| 131,072 | 121 / 139 | 599.9 | 25.7 | 3.6 min | EOS |
| 262,144 | 114 / 133 | 439.9 | 21.3 | 9.9 min | EOS |
| 524,287 | 98 / 120 | 296.1 | 15.4 | 29.5 min | EOS |
| 1,032,192 | 67 / 96 | 179.6 | 9.6 | 95.8 min | EOS |
Every rung returned a complete, correct answer to the same five-function coding task buried behind the context.
The other per-metric charts: time to first token,
MLX allocator peak, process footprint peak,
completion tokens. The tables with rows per rung and the provenance of every cell are in the
plugin's docs/benchmarks/mimo-v26-flash/README.md.
HumanEval (164 problems, one sample each, pass@1, rows 127 / 145; greedy = temperature 0, thinking off, 8,192-token cap; card = temperature 1.0, top_p 0.95, seed 1, thinking on, 16,384-token cap): greedy 155 / 164 = 0.9451, card 157 / 164 = 0.9573; every failure is a wrong answer, none truncated, looped, without code or errored.
Why a dynamic expert cache: within one context the busiest half of a layer's experts takes about 85% of the routes, but which half changes with the context, so the plugin's dynamic residency (59% of the bank resident) hits 94.5% of decode routes where a static hot set of the prompt's top experts would hit 50-58%.
Provenance
Built from XiaomiMiMo/MiMo-V2.6-Flash-MOPD at revision 2479e2d0029eca9a34cc7e7f55a121925f81908e: the 128 expert
shards (171,743,035,008 bytes, each sha256-checked against its LFS object) repacked record by record into
experts.bin (parity-checked bit-exact on sampled records; every record's sha256 is in expert-manifest.json), the
residents converted to one safetensors file with the release's FP8 projections kept, the MTP heads cut into the
sidecar. config.json, the tokenizer files and the chat template are the release's with the expert_streaming
block added. The build steps are listed in the plugin's docs/mimo-pack.md.
Measurements and software: mlx-serve host pr/mimo-host (2c041af6) with the mlx-stream plugin at 05fc760
(binary sha256 4b018c54...); the ladder and HumanEval ran on the predecessor build (plugin 8e7d186), which the
binary of record reproduces token for token on the 16K cell. Charts: scripts/mimo_charts.py in the plugin.