DeepSeek-V4.1-Flash, streaming EXL3 packs for mlx-serve
Each branch of this repo is a streaming EXL3 pack of the earlier base,
deepseek-ai/DeepSeek-V4.1-Flash at revision
dba1be0a40aa45a94ad051997016db3960a90277, for mlx-serve. The 3.0bpw branch
carries Mia-AiLab's EXL3 3.0 bpw quantization of the routed experts, and mlx-serve serves it with the
mlx-stream plugin. The trunk loads once, and the routed experts and the Engram
tables are read from SSD as they are needed, so a 128 GB Mac can serve the model. Each bitrate is a branch, and main
holds only this card and the license. For the refreshed 1017 weights, see
OpensourceWTF/DeepSeek-V4.1-Flash-1017-streaming-repack-exl3.
Branches
| Branch | bpw | Size | KLD / PPL | Source and credit | Status |
|---|---|---|---|---|---|
3.0bpw |
EXL3 3.0 bpw routed experts (K3 in every layer); the other weights unchanged in value | 426.4 GB | not measured | Mia-AiLab's EXL3 3.0 bpw quantization, Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw (revision c5534b90), made with ExLlamaV3 v1.4.2 by turboderp, in our streaming layout |
live |
Sizes are in GB, where 1 GB is 10^9 bytes.
Related repositories
| Repo | What it holds |
|---|---|
| OpensourceWTF/DeepSeek-V4.1-Flash-1017-streaming-repack-exl3 | The refreshed 1017 weights in the same streaming layout, a branch per bitrate, with their quality and speed data. |
| OpensourceWTF/DeepSeek-V4.1-Flash-1017-exl3-2.81bpw | An EXL3 SC 2.81 bpw quantization of the 1017 weights. It holds exllamav3's own files and is the source of the 1017 repo's 2.81bpw branch. |
How to download and run
Download the branch with:
hf download OpensourceWTF/DeepSeek-V4.1-Flash-streaming-repack-exl3 --revision 3.0bpw --local-dir DeepSeek-V4.1-Flash-streaming-repack-exl3-3.0bpw
The pack is served by mlx-serve with the mlx-stream plugin (deepseek_v41 arch), which is proposed for mlx-serve in
ddalcu/mlx-serve#749. Build mlx-serve with the mlx-stream plugin,
following the plugin's README and PR #749, then start the server:
mlx-serve --serve --model /path/to/DeepSeek-V4.1-Flash-streaming-repack-exl3-3.0bpw --port 8080 \
--memory-ceiling-gb 120.259 --wired-margin-gib 2
--memory-ceiling-gb 120.259 is 112 GiB, which leaves macOS 16 GiB of a 128 GiB Mac, and the plugin admits its expert
rows against that ceiling. --ctx-size sets the largest prompt the server bills for at load, and without it the server
bills for 16K prompts. A larger context leaves fewer expert rows resident. The mlx-serve app offers prompts of 1K, 2K,
4K, 8K, 16K, 32K, 64K, 128K, 256K and 512K tokens, each plus a 1,088-token reply allowance, and the model's limit of
1,048,576 tokens. The download is about 426 GB.
Other loaders, such as Transformers, vLLM or ExLlamaV3, do not read this layout. For those, use the original release or the EXL3 quantization linked above.
Quality
KLD was not measured for this pack.
The 3.0 bpw EXL3 routed experts are lossy compared with the release's FP4 experts, and they are the only lossy part of the repack. Everything else matches the release value for value: the resident projections, MTP experts and Engram tables are exact repacks, and the BF16/F32 tensors are verbatim. The served decode uses DSpark speculative decoding with typical acceptance and rounding-class kernels, so its output is close to the BF16 reference but not bit-identical.
HumanEval
HumanEval was served through mlx-serve with the flags above, with one greedy sample per problem (temperature 0). It was scored offline by running each problem's tests on the last fenced code block the model committed to.
| Quantity | Value |
|---|---|
| pass@1 | 0.9634 (158 of 164) |
| Problems failed | HumanEval/32, 83, 132, 140, 145, 163 |
| Completions truncated / errored | 0 / 0 (all 164 finished with a stop) |
| Completion tokens | 23,939 total, mean 146, max 4,073 |
| Decode tok/s | 31.8 aggregate; per request median 30.0, p10 20.2, p90 36.0 |
| Time to first token | median 7.65 s, p90 9.08 s (prompts of about 200 tokens) |
| Speculative acceptance | 4.89 tokens accepted per draft attempt |
| Wall time | 2,077 s for 164 requests; peak process memory 108.3 GB |
The same 158 of 164, with the same six failures, was scored on the previous build of the server, so the score is stable across builds. The six failures are the model's at this quantization, not harness errors.
Speed
The speed figures were measured on an Apple M5 Max with 128 GiB, served through mlx-serve with the flags above. The build was mlx-stream plugin 4f42e79. The ladder was measured on mlx-serve host 72b36513 (server sha256 c8cbf14e), and the shipping host 3296dc0e, which has upstream main 84c6ee64 merged, reproduces the 16K headline token for token.
On a 16K-token prompt, the pack ran at 661.5 prefill tok/s, with a TTFT of 24.77 s, and 36.8 decode tok/s. The server admitted 135 prompt rows and 166 decode rows, and its peak was 107.0 GB.
Prefill tok/s is prompt tokens ÷ prefill time, and TTFT is the seconds from request to first output token. Decode tok/s is generated tokens ÷ decode time. Prompt rows and decode rows are the expert slot rows per layer that the server admits for the prompt pass and for decode. Peak GB is the server process's peak memory, in decimal GB.
Prompt-size ladder
The ladder runs from 1K to 1M tokens, one cold request per size on a fresh server (max_tokens 1,024, temperature 0), measured in one window on this build. The memory bill was derived per term: the attention selection's indices were carried, and the score chain was billed at what the chunked selection holds, so long prompts admit more expert rows.
| Prompt tokens | Prefill tok/s | TTFT (s) | Decode tok/s | Prompt rows | Decode rows | Peak GB |
|---|---|---|---|---|---|---|
| 1,024 | 81.3 | 12.6 | 39.9 | 148 | 167 | 107.2 |
| 2,048 | 138.3 | 14.8 | 39.7 | 142 | 166 | 106.8 |
| 4,096 | 264.4 | 15.5 | 38.3 | 137 | 166 | 106.8 |
| 8,192 | 461.5 | 17.8 | 38.6 | 137 | 166 | 106.8 |
| 16,384 | 667.5 | 24.6 | 37.9 | 135 | 166 | 107.0 |
| 32,768 | 618.5 | 53.0 | 37.9 | 135 | 166 | 107.1 |
| 65,536 | 537.5 | 121.9 | 35.5 | 135 | 165 | 106.9 |
| 131,072 | 501.2 | 261.5 | 32.5 | 130 | 164 | 106.8 |
| 262,144 | 456.3 | 574.5 | 27.1 | 115 | 161 | 106.4 |
| 524,288 | 379.5 | 1,381.5 | 18.2 | 91 | 155 | 106.3 |
| 1,047,488 | 262.6 | 3,988.9 | 11.1 | 82 | 143 | 103.7 |
Fewer rows fit beside a larger KV, and decode slows with depth as each cycle attends over more positions. The 16,384 row is the ladder's own cell, while the 16K headline above (661.5 / 24.77 s / 36.8) was measured in its own run. The context limit is the model's 1,048,576 tokens.
How the pack was made
The 3.0bpw pack copies Mia-AiLab's EXL3 3.0 bpw routed experts byte for byte into a layout that streams them from
SSD. It does not re-quantize anything, and it converts the other weights to MLX's native float formats as exact
repacks. The
3.0bpw branch card
describes every file and how the expert bank was checked.
Credits
DeepSeek made the model and released it under the MIT license (LICENSE). Mia-AiLab (MiaAI Lab) made the EXL3 3.0 bpw
quantization of the routed experts,
Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw (MIT),
with ExLlamaV3 v1.4.2 by turboderp.
The server is mlx-serve by David Dalcu, and the plugin is mlx-stream. The repack is by David Tai.
License
DeepSeek released DeepSeek-V4.1-Flash under the MIT license, and the LICENSE file in this repo is that license.