DeepSeek-V4.1-Flash-1017: EXL3 3.0 bpw for 2× RTX PRO 6000 Blackwell (diffbot layout)

This is DeepSeek-V4.1-Flash-1017, the October 17 update of DeepSeek-V4.1-Flash, packed in the layout of diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000 (the "diffbot pack"). It replaces the weights of that pack and runs on diffbot's recipe unchanged, at the same speed and memory footprint: two 96 GB cards, about 230 GiB of host RAM, 1M-token context.

This is a repack, not a new quantization of the routed experts. The routed experts are PixelML's uncalibrated EXL3 3 bpw quant. What this repo adds is the rest of a servable pack for the 1017 weights: the DSpark drafter and the Engram tables, re-encoded from DeepSeek's 1017 files with diffbot's scripts, plus measurements.

What's in this repo

Why a repack is needed: 1017 has the same architecture and tensor names as V4.1-Flash, but DeepSeek retrained every weight, so all 48 files changed, including the Engram tables and the drafter. The diffbot pack's files can't be mixed with 1017 files. PixelML's repo has the 1017 experts but no Engram tables and an MXFP4 drafter.

Quick start

You need two sm_120 GPUs with 96 GB each, Docker with the NVIDIA container toolkit, about 250 GiB of free host RAM and 340 GB of disk (the same as the diffbot pack).

# 1. This pack
hf download Nateemerson/DeepSeek-V4.1-Flash-1017-EXL3-3bpw-2x-RTX-PRO-6000 --local-dir ~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw

# 2. diffbot's recipe, at the revision tested here (the recipe only, not the old weights)
hf download diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000 --revision a1cf1bd227a7 \
  --include "recipe/*" --local-dir ~/dsv41-recipe
cd ~/dsv41-recipe/recipe
bash docker/build.sh       # serving image dsv41-flash-exl3-3bpw-sm120
bash kernels/build.sh      # the xmoe and xmoe_do MoE kernels

# 3. Serve: OpenAI-compatible API on :8000, model name deepseek-v4.1-flash
PACK=~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw PLAN=plan-blend30-dh.json API_KEY=change-me bash serve/serve.sh

• PACK must point at this repo's folder, because the recipe isn't inside it.
• PLAN=plan-blend30-dh.json is what every number below used. The recipe's default is plan-agent30-dh.json.
• On our host we also set NCCL_P2P_DISABLE=1 (GPU-to-GPU P2P hangs there), CONTAINER_MEM=236g (the host has 249 GiB; the default is 260g) and EXTRA_ARGS="--host 127.0.0.1". Leave P2P on if it works on yours.
• diffbot's newer recipe revision (6062be5f, rebased onto a newer vLLM nightly) was not tested with this pack.

Speed

Same machine, same image, same recipe, same plan. Only the weights differ between the two columns.

• Speed matches the old pack within run-to-run noise. The one visible difference: at 4 streams the retrained drafter's guesses are accepted a little less often (3.48 vs 3.67 tokens per step).
• These are lower than the diffbot card's headline numbers. That card measured a different setup: the gen30 plan (faster on synthetic prompts) on 2× Max-Q cards at 300 W. Ours ran the blend30 plan with P2P off.
• Memory: VRAM 91.5 / 90.4 GB on the two cards (the old pack: 90.7 / 90.3). Lowest free host RAM while serving: 12.8 GB.

Test machine:

• 2× RTX PRO 6000 Blackwell Workstation Edition (96 GB), power-limited to 450 W, PCIe 5.0 x8 each, P2P off.
• AMD Ryzen 9 9950X, 256 GB DDR5 (249 GiB usable), NVMe.
• Driver 610.57.04; image dsv41-flash-exl3-3bpw-sm120 built from diffbot's recipe at a1cf1bd2.

Quality

Both packs ran diffbot's quality probe (recipe/bench/quality-probe.py: GSM8K first 200 and HumanEval 164, greedy, thinking off, 4 streams, max 4,096 tokens) on the same machine and setup. Only the weights differed.

• The two packs are within one item of each other on both tests, and the diffbot pack's run reproduces its published numbers, so the setup matches.

• Item by item:

  • GSM8K: 1017 solves 2 problems the old pack misses and misses 1 it solves.
  • HumanEval: 1017 passes 3 the old pack fails and fails 4 it passes.

  Both tests are close to their ceiling for this model, so they show no damage from the uncalibrated experts but can't rule out a small loss.

• HumanEval ran sandboxed. Each program ran in a fresh container: no network, read-only filesystem, unprivileged user, 512 MB, 15 s. That replaces the probe's direct execution on the host; the rest of the script is unchanged. All 8 failures were wrong answers, not sandbox limits.

• Raw per-item outputs, including the generated code, are in results/probe-1017-pack.jsonl and results/probe-diffbot-pack.jsonl.

What hasn't been measured:

• KL divergence against the original model. The reference logits for KL need the 510 GB original, which can't run on this machine, and nobody has published a 1017 reference set yet. The diffbot pack measured KL 0.054 / 0.032 / 0.010 (neutral / code / agentic) for V4.1-Flash with calibrated experts. Expect this pack's to be somewhat higher, because PixelML's experts are uncalibrated.
• DeepSeek's claimed gains for 1017 (Terminal-Bench, DeepSWE and others). These are full-precision numbers and weren't checked here.

Limitations

• Uncalibrated routed experts (identity Hessian). When a calibrated 1017 EXL3 3 bpw quant appears (coolbho3k made the calibrated one for V4.1-Flash), swapping it into shards 1-43 should be an upgrade.
• The host-expert plans come from the old model. They decide which 30% of the experts live in host RAM, and were built from V4.1-Flash routing statistics. A 1017-specific plan (EXL3_EXPERT_STATS + recipe/plans/make_plan.py) may decode faster.
• Tested only with diffbot's recipe at a1cf1bd2, on one machine.
• The drafter and Engram encodings are lossy, the same as in the diffbot pack: Engram first-chunk relative error 0.089 (the script measured 0.093 on V4.1-Flash).

How it was built

All steps follow diffbot's recipe/quantize/README.md, with diffbot's scripts unchanged. build/ has the assembly script; the times are from this machine.

1. Download PixelML @ 3d095b72 (all files) and DeepSeek 1017 @ 1bae0e10 (shards 44-48, config, index, tokenizer, encoding/). Every file used was checked against the Hub's sha256.

2. Drafter: quantize_mtp_exl3.py --bits 3 --greedy --beam 16 --batch 8 on shards 44-46, on one GPU: 1,152 matrices in 419 s. quantize_mtp_exl3.py imports tools/pack_meta.py from sfxnz's recipe at commit 460ac89c, the commit diffbot's quantize_mtp_exl3.diff applies to.

3. Engram: convert_engram_int4.py on shards 47-48: CPU only, about 9 minutes.

4. Assembly: build/assemble_pack_1017.py. It is diffbot's assemble_pack.py adapted to PixelML's shard layout:

   • PixelML shards 1-43, then the new drafter (44-46) and Engram (47-48) shards;
   • the index is rebuilt from the shard headers, because PixelML's index lists .mcg tensors its shards don't contain;
   • config.json is the diffbot pack's, with only quantization_config.source changed.

   The result has exactly the diffbot pack's 190,549 tensor names, and its drafter and Engram tensors have the same dtypes, shapes and sizes as the diffbot pack's.

Related work

• cliecy/DeepSeek-V4.1-Flash-1017-2xRTXPRO6000: an independent recipe for the same stack, published the same night. It runs without Docker (bubblewrap), keeps the drafter in MXFP4, and reports GSM8K, MATH-500, HumanEval, MMLU-Pro and C-Eval results.
• LibertAIDAI published calibrated 1017 EXL3 quants (2.9 and 3.5 bpw). They quantize every layer in ExLlamaV3's native layout, so they don't load in this stack.

Credits and license

Built entirely on the work of:

• DeepSeek-AI: the model and weights (DeepSeek-V4.1-Flash-1017, MIT).
• PixelML: the 1017 routed-expert quantization (PixelML/DeepSeek-V4.1-Flash-1017-EXL3-3bpw, MIT).
• diffbot: the pack layout, the serving recipe, and the drafter, Engram and assembly scripts (diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000, MIT).
• sfxnz: the V4.1 EXL3 quantization tooling (sfxnz/DeepSeek-V4.1-Flash-EXL3-vLLM-2x-DGX-Spark, MIT).
• turboderp: ExLlamaV3 and the EXL3 format (turboderp-org/exllamav3, MIT).
• coolbho3k: the calibrated V4.1-Flash quant behind the diffbot pack that this one is measured against.

The weights keep DeepSeek's MIT license (LICENSE). Repacking adds no restrictions.