DeepSeek-V4.1-Flash-1017 · EXL3 3.5 bpw

TL;DR — EXL3 quantization of deepseek-ai/DeepSeek-V4.1-Flash-1017 at 3.53 bpw (239.9 GiB). On held-out text its perplexity is +0.8% (code) / +0.1% (prose) above the unquantized model. It uses the same tensor layout as Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw and the 2.9 bpw sibling, with higher bitrates. Serving it with the MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks kit needs the exl3_k_map.json shipped here. The two Engram tables are not in this repo: take them from the 1017 base repo, because they changed in this release.

Does not fit 2× DGX Spark. At TP=2 this build needs 121.5 GiB of weights per rank, more than a DGX Spark has available (~117 GiB). On two Sparks use the 2.9 bpw build (~98 GiB per rank).

Chat template: this repo ships chat_template.jinja, a Jinja port of DeepSeek's 1017 encoding/encoding.py (from jayleaton's MIT template via a community PR by dpolist, amended for the 1017 encoder). It renders byte-identically to the reference on our 8,455-case test corpus. Runtimes that require a template file at startup need it (verified to render identically under Python Jinja and TensorFold's Zig engine); vLLM-based kits use their built-in DeepSeek encoder.

Agentic use / reasoning effort: with tools in the conversation, each turn's reasoning stays in the prompt and is re-read by every later turn, so reasoning cost compounds. On our benchmark of the original 1017 weights (3,738 jobs: agentic coding, tau-bench, LiveCodeBench, AIME, chat), reasoning_effort high (75, the default) matched max (100) on quality at 0.55–0.8× the cost, and low (50) matched too at 0.4–0.6× (results in discussion #4). Avoid max for agent loops unless you need the headroom.

Tensor groups

Every EXL3 matrix is stored as trellis (int16) + suh / svh (fp16) + mul1 (int32 codebook marker).

Perplexity

Two held-out sets of unpublished text, 32 sequences x 2048 tokens each, tokens 2-2048 of every sequence scored (65,504 tokens per set). Every row uses the same full-sequence forward (see How it was made).

Engram tables (required)

DeepSeek-V4.1 adds two Engram n-gram lookup tables (layers.1.engram.embed, layers.14.engram.embed: FP8 with E8M0 block scales, ~95 GiB each). They are used unquantized and come from the base checkpoint, like in the existing V4.1 EXL3 serving stacks. The 1017 tables are new: the shards from the older deepseek-ai/DeepSeek-V4.1-Flash must not be reused.

hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --include model-00047-of-00048.safetensors model-00048-of-00048.safetensors model.safetensors.index.json config.json \
  --local-dir ./engram-src-1017
sha256sum engram-src-1017/model-0004[78]-of-00048.safetensors

Serving

The checkpoint follows the layout of Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw, so it loads with the vLLM-based MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks kit. That kit targets 2x DGX Spark at TP=2, where this 3.5 bpw build does not fit (121.5 GiB of weights per rank); use the 2.9 bpw build there. The settings below come from the kit's start.sh / download.sh (commit 6f7d159). Upstream exllamav3 has no DeepseekV41 model class yet, so it cannot load this checkpoint directly.

Fresh install (on the head node, from the kit checkout):

git clone https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
cd DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
cp .env.example .env
cat >> .env <<'EOF'
HF_MODEL_REPO=LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw
MODEL_HOST=$PWD/model-1017-3.5bpw
EXPECTED_SHARDS=44
HF_ENGRAM_REPO=deepseek-ai/DeepSeek-V4.1-Flash-1017
ENGRAM_DIR=./engram-src-1017
KMAP_HOST=$PWD/model-1017-3.5bpw/exl3_k_map.json
EOF
./download.sh   # run first: start.sh checks KMAP_HOST before it fetches the weights
./start.sh

.env is sourced by bash, so $PWD expands to the kit directory when the scripts are run from it.

KMAP_HOST is required here: the per-layer expert bitrates differ from the kit's bundled files/exl3_k_map.json, and the kit sizes each MoE layer's packed tensors from that map.

Switching an existing V4.1 install — the kit reuses anything already on disk, so make sure nothing old gets picked up:

How it was made

Smoke test outputs (greedy decoding)

chat: What is 17*19? Reply with the integer only.

323<｜end▁of▁sentence｜>

chat: Explain in three sentences why the sky is blue.

Sunlight contains all colors, but air molecules scatter short blue wavelengths far more than long red ones—a process called Rayleigh scattering. This scattered blue light reaches your eyes from every direction, making the sky appear blue. At sunset, light travels through more atmosphere, so the blue is scattered away and reds and oranges dominate.<｜end▁of▁sentence｜>

thinking: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?

We need answer classic. Need be careful. Let ball x, bat x+1, total 2x+1=1.10 => x=.05. Answer 5 cents. Need concise.</think>The ball costs **$0.05 (5 cents)**.

The bat costs $1.05, so together they total $1.10.<｜end▁of▁sentence｜>

Files

Credits

• DeepSeek for deepseek-ai/DeepSeek-V4.1-Flash-1017 (MIT).
• turboderp for exllamav3 (MIT): quantizer, kernels and conversion pipeline.
• Mia-AiLab for Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw and the MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks kit, whose tensor layout this release follows (the per-tensor bitrates differ, so the kit needs this repo's exl3_k_map.json).

License

MIT, same as the base model.