DeepSeek-V4.1-Flash-1017, streaming EXL3 packs for mlx-serve

Each branch of this repo is a streaming EXL3 pack of deepseek-ai/DeepSeek-V4.1-Flash-1017 at revision 1bae0e1088993b9b82de08f59de5be5a34e42170, laid out for the streamed experts of mlx-serve. The 2.81bpw pack is our EXL3 SC quantization, made with turboderp's exllamav3 SC pipeline and served by mlx-serve's turboexl3_ds41 plugin. The 2.9bpw pack carries Mia-AiLab's EXL3 EAQ v1 2.9 bpw quantization and is served by mlx-serve with the mlx-stream plugin. The 1.68bpw, 2.25bpw and 2.68bpw packs are turboderp's own EXL3 quantizations in the same layout, and they are for testing until a turboexl3_ds41 build that serves their fractional layers is published. In every pack the trunk loads once, while the routed experts and the Engram tables are read from SSD as they are needed, so a 128 GB Mac can serve the model. Each bitrate is a branch, and main holds this card, its charts (assets/) and the license.

Branches

Sizes are in GB, where 1 GB is 10^9 bytes. PPL and KLD are measured against the release's own weights, whose PPL is 1.7730. Quality has the full data and how it was measured, and Speed has the speeds.

Related repositories

How to download and run

To download one branch, pass its name to --revision. For example, this command downloads 2.81bpw:

hf download OpensourceWTF/DeepSeek-V4.1-Flash-1017-streaming-repack-exl3 --revision 2.81bpw --local-dir DeepSeek-V4.1-Flash-1017-streaming-repack-exl3-2.81bpw

mlx-serve serves 2.81bpw with the turboexl3_ds41 plugin from CommunityServe/plugins PR #24. The branch's a4k expert bank needs the plugin's turboexl3-zzz8m-a4k patch or later, and earlier builds refuse the bank at load. Its one-read Engram records need turboexl3-zzz8t-engram-records or later, and earlier builds read the same records as two reads a row.

mlx-serve serves 2.9bpw with the mlx-stream plugin. The 2.9bpw branch card gives the plugin build it needs and the command that starts the server.

mlx-serve serves 1.68bpw, 2.25bpw and 2.68bpw with the turboexl3_ds41 plugin, but only in a build with its fractional-layer support (MLX_SERVE_DSV41_FRAC), because some of their routed layers use exllamav3's half-integer rates of 1.5 and 2.5 bits per weight. That support is in the draft PR CommunityServe/plugins#35, which is stacked on PR #24, so these three branches are for testing until a build with it is published. Each branch's card gives the details.

Quality

The PPL and KLD figures come from one run of exllamav3's qbench. It compared the release's own FP8/FP4 weights, run in exllamav3, with each quantized pack, on the same tokens and against the same reference. The KLD was measured with exllamav3's engine, so mlx-serve's runtime, for example FLASH_ATTN's prefill, was not part of it.

The 2.81bpw figures were measured on the raw exllamav3 pack, OpensourceWTF/DeepSeek-V4.1-Flash-1017-exl3-2.81bpw. The branch holds the same quantized weights, re-laid for streaming.

The 2.9bpw figures were measured on its EXL3 source, Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw at revision 45ee5d0f. The branch holds that repo's routed experts byte for byte, while its other weights come from the release as exact MLX repacks, not from that repo. That repo has no Engram tables, so its run used the same release FP8 tables as the 2.81 bpw pack.

The 1.68bpw, 2.25bpw and 2.68bpw branches were not measured on this test set. turboderp's model card has his own KLD and perplexity charts for them, and his AIME-25 and GPQA-Diamond scores.

The test set is a held-out set of 20 conversations, built from turboderp's general prompts with 30% of them using tools, at medium reasoning effort. It holds 7,109 prompt tokens and 45,434 completion tokens, and FP8 hosts of the original model generated the completions. The calibration data was a separate trace of 217 rows.

KLD is KL(release ‖ pack) in nats over the completion tokens, with the full 129,280-token vocabulary. The noise floor is the release itself with BF16-scale rounding noise added after every layer, and every pack's KLD includes it.

KLD by the release's confidence in its top token

The chart shows the mean and the median KLD of each pack in each bucket of the release's top-token probability, with each bucket's share of the tokens, on a log scale.

The token shares are the same for both packs, and the method is the one described above.

The paired difference, the 2.81 bpw pack minus EAQ v1 2.9 bpw, is -0.00048 KLD (-1.2%), with a 95% CI of -0.0025 to +0.0016 from a bootstrap over the 20 conversations. The 2.81 bpw pack is lower on 10 of the 20 conversations. The result is a tie in quality, at about 2.7% fewer bytes of main model weights for the 2.81 bpw pack.

Mia-AiLab's own published figures

Mia-AiLab's model card reports the 2.9 bpw pack's fidelity on its own test sets, measured in its own exllamav3 port. On short context, 100 conversations of third-party text with 89,880 scored tokens, it reports KL 0.0137 nats (95% CI 0.0115 to 0.0165) and a top-1 agreement of 96.6%. On long agentic sessions, 8 synthetic sessions of 8-31k tokens, it reports KL 0.0051 to 0.0060.

Those numbers are not directly comparable with the tables above, because the test text is different. Mia-AiLab's sets are third-party conversations, while the set above is the model's own reasoning traces. On the set above, the same EAQ pack measures 0.0401, so that set is about three times harder for both packs. Compare packs within one table, not across them.

Speed

These figures come from the calibrated sweep, which ran 2.81bpw and turboderp's three packs at nine prompt sizes, from 1K to 256K tokens. Each figure is the median of 4 fresh servers, in tok/s. The 2.9bpw branch was not part of the sweep. Its branch card records one served smoke test on mlx-stream's dsv41-exl3-k2 build.

Cold prefill (tok/s)

Greedy decode (tok/s)

Card decode, sampled (tok/s)

¹ At 256K, 2.25bpw's sampled text ran 1,276 tokens, against 516–764 tokens at every other point, so its 256K card decode figure is not comparable with the others.

How the sweep was measured

The sweep ran on an Apple M5 Max with 128 GB under macOS 27.2, with the fans at maximum. The server was mlx-serve 1271abe0 with its host patches (G_KITBM, 46bbf9bd8b4e), and it ran the turboexl3_ds41 plugin with its calibrated memory bill. 2.81bpw ran on plugin build ctl60 (dylib 4a99f8fffb2f), and turboderp's three packs ran on tdp5 (dylib 2d9e2cb095e9), which is the same plugin with fractional-layer support.

At each size, the prompt was the same code task, built up to that length. Each fresh server ran a keyed sampled request, the card, at temperature 1.0, top-p 0.95 and top-k 20, and then a greedy request, on the same seeds, with up to 16,384 output tokens. Every request stopped on its own, and the DSpark verify policy was off. Cold prefill is the card request's prompt processing on its fresh server, and the two decode figures are the two requests' decode speeds.

Each figure is the median of 4 fresh servers. Every text was identical across the 4 servers, and each greedy text was equal to its 1-row reference.

Memory

The plugin's calibrated memory bill predicts each server's peak memory before it loads, and in the sweep every server's peak landed within −0.13 to +0.20 GiB of that prediction. Panel (a) adds what the rest of the machine held when each window started: the whole machine stayed at or under 110.5 GiB, the bill's target, apart from brief spikes. Those spikes came from macOS's Apple Intelligence on-device model service, which took up to 2.4 GiB at a time, and the plugin is getting an allowance for it. Panel (b) shows how many expert-cache rows each pack kept as the prompt grew, as a share of its rows at 1K.

Parity of the plugin port

The head of CommunityServe/plugins PR #24, commit 0745d79, serves the same texts as the measured build, and it is within 0.5% of its speed. These are the 16K parity windows on 2.81bpw, pcp for prefill and pcd for decode, in tok/s:

Both arms ran on one server, mlx-serve 1271abe0 with its host patches (G_KITBM), on the same Mac under macOS 27.2, with the calibrated bill on and pinned per window. Arm a is plugin build ctl60, the build the sweep measured, and arm b is the PR #24 head, built against mlx-serve db128461's kit. The prefill cells are the 16K-token code prompt, cold on a fresh server, and then a 14,917-token slice of new text on the same server. The decode cells are the 16K card, keyed and sampled, and then greedy, with the DSpark verify policy off.

The prefill window ran a warm server (a) and c1 (b) first, then a1 b1 b2 a2. The decode window ran w0 (b) first, then a1 b1 b2 a2 b3 a3 a4 b4. The b − a column is the b mean against the a mean. The texts were identical in every a and b arm of both windows, with the same bank rows (149 prompt, 186 decode) in every arm.

The port also ran on stock mlx-serve db128461, one server in each window, without the three host patches of the draft PR ddalcu/mlx-serve#1065. Without them, BANK_POOL is off, MLX uses its own command-buffer split and the host uses its default sampler. Against the a arms, the stock server measured:

The default sampler drew different tokens after the prefill prompts and in the sampled decode, so the sampled decode is not a like-for-like speed. Greedy decode produced the same text.

How the packs were made

2.81bpw

2.81bpw is repacked from the raw exllamav3 quant OpensourceWTF/DeepSeek-V4.1-Flash-1017-exl3-2.81bpw, which was made with turboderp's exllamav3 SC pipeline. The branch holds these parts:

Each expert record's three trellis codes start on a 4 KiB block. That layout is a4k, declared in expert-manifest-v2.json, and it lets the plugin read the codes straight into its expert slots.

Each Engram record holds a row's 256 E4M3 codes, then its 8 E8M0 scales, so one n-gram lookup is one read. Every row was checked byte for byte against the release's tables.

The trunk's quantized layers are EXL3 K6 with the MUL1 codebook, except the routed experts of the three DSpark (MTP) stages, which are Mia-AiLab's 4-bit EXL3 tensors, byte for byte. A few tensors of the vision tower and aligner are kept in BF16, and the router, norms, hyper-connections, sinks and embeddings are as released.

The plugin reads expert-usage-v1.json at load to choose which experts start resident. The file changes no outputs, and MLX_SERVE_DSV41_RESIDENT_SEED=none turns it off.

The branch leaves out ngram_embedding.safetensors, the release's Engram tables in their original layout (202.8 GB). mlx-serve reads engram/ whenever engram/engram-manifest.json is present. config.json still names the original file under omlx_deepseek_v41.engram_tables, but that entry is read only when the manifest is absent. The original layout is in OpensourceWTF/DeepSeek-V4.1-Flash-1017-exl3-2.81bpw, with exllamav3's own files.

2.9bpw

2.9bpw holds Mia-AiLab's EAQ v1 2.9 bpw quantization of the 1017 weights, Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw at revision 45ee5d0f, in our streaming layout. Its routed experts are that repo's tensors byte for byte, at K3 with K2 on layers 18-22. Its other weights are unchanged in value and come from the release as exact MLX repacks. The 2.9bpw branch card lists every file and how it was built and checked.

Credits

DeepSeek made the model and released it under the MIT license (LICENSE). turboderp created exllamav3, the EXL3 format and the SC quantization pipeline, which produced the 2.81 bpw experts and trunk. Mia-AiLab (MiaAI Lab) made the EXL3 EAQ v1 2.9 bpw quantization DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw (revision 45ee5d0f), which supplies the routed experts of 2.9bpw and the 4-bit EXL3 DSpark stage experts in 2.81bpw.

The server is mlx-serve by David Dalcu. 2.81bpw runs on the turboexl3_ds41 plugin in CommunityServe/plugins, built on turboexl3, and 2.9bpw runs on the mlx-stream plugin. The repack is by David Tai.

License

DeepSeek released DeepSeek-V4.1-Flash-1017 under the MIT license, and the LICENSE file in this repo is that license.