DeepSeek-V4.1-Flash-1017 · dsx 2-bit

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) in 148 GB, built to run on two RTX 3090s and 128 GB of RAM with exact routing. Runs on dsx, a Rust + CUDA engine made for this model. For the best quality on the same machine, use dsx-ud (KL 0.269, one 305 GB download).

Which build

All four run on dsx and were measured on the same eval (Lucebox 1017, 8 x 1,024 tokens against the FP8 model's top-256 log-probs).

At a glance

Quality

Measured on Lucebox's 1017 eval set: 8 × 1024 tokens against the FP8 model's top-256 log-probs.

Unsloth's GGUFs were measured in llama.cpp with the same token ids and metric (tool). They also carry the Engram table, quantized to IQ4_NL (110.6 GB); dsx reads it unquantized from the official shards.

Reasoning

Calibrated on the full model's own reasoning traces as well as agent text. Reasoning tokens per run on two Omni-MATH problems, temperature 1.0, reasoning effort low, all answers correct:

On harder problems it reasons longer than the FP8 model, up to about 2×. Sample at temperature 1.0 for reasoning.

Speed

Greedy decoding, exact routing, 400 generated tokens. The RTX 3090 numbers come from build v4, which has the same expert layout as this build (v4b); the RTX 5090 numbers come from the older build v2.

On the same 2x RTX 3090 machine, Unsloth's UD-IQ1_M in llama.cpp (PR #30817) runs at 12.0 tok/s decode and 75-109 tok/s prefill on the 3K-token agent prompt.

Decode speed comes down to how many experts fit in VRAM + RAM. Two 24 GB GPUs + 128 GB hold all of them; one 32 GB GPU

• 128 GB leaves ~40 GB of experts to be read from disk.

Size

Run

This build needs dsx from 2026-10-19 or later. Older builds do not know the rotation and produce garbage with it: pull and rebuild dsx first.

# the quant: 4 GGUF shards, loaded as one model
hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-2bit --local-dir dsx-2bit

# Engram tables + tokenizer from the official checkpoint
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 \
  --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json

# the engine (dsx's NVIDIA runner)
git clone https://github.com/lxe/dsx && cd dsx
CUDA_ARCHS=86 cargo build --release -p dsx --features cuda      # 86 = RTX 30xx, 120 = RTX 50xx

./target/release/dsx-server --model ../dsx-2bit/DeepSeek-V4.1-Flash-1017-dsx-2bit-00001-of-00004.gguf \
  --engram ../v41-1017 --tokenizer ../v41-1017/tokenizer.json --host 0.0.0.0 --port 8090 \
  --ctx 131072 --arena-gb 110 --rank ../dsx-2bit/expert-rank.json

Open http://localhost:8090/ for the web UI, or point any OpenAI-compatible client at http://localhost:8090/v1:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8090/v1", api_key="none")
r = client.chat.completions.create(model="deepseek-v4.1-flash", reasoning_effort="none",
                                   messages=[{"role": "user", "content": "Merge overlapping intervals in Python."}])
print(r.choices[0].message.content)

Thinking is on by default; "reasoning_effort": "none" turns it off. Images go in as data URLs (image_url), or as base64 image blocks on the Anthropic endpoint; the web UI takes uploads.

Hardware

VRAM + RAM should hold ~140 GB of experts:

How it was made

• Experts: each of the 15,360 routed experts gets ternary (1.75 bpw), Q2_0 (2.25) or Q3_K (3.44), picked by its measured output error. One bit budget is shared across layers 2-36, so sensitive layers get up to 2.75 bpw and forgiving ones ~1.76. Average: 2.03 bpw.
• Method: GPTQ per expert, with Hessians from a layer-by-layer calibration pass through DeepSeek's reference model.py (GSQ pipeline from Xolotl).
• Rotation: every routed expert is quantized in a Hadamard-rotated basis (128-wide blocks along each input dimension), which spreads outlier weights before rounding; dsx rotates the expert inputs to match at run time (GGUF key dsx.expert_rot). Same size as without it, KL 0.397 → 0.382.
• Scale refit: after GPTQ picks each expert's codes, its group scales are refit by least squares against the full calibration Hessian, GPTQ runs again from the refit scales, and a final refit follows. KL 0.382 → 0.373.
• Calibration text: 160 × 2048 tokens: coding-agent sessions, antirez's agent and code prompts, and 53 sequences of the full model's own reasoning traces on Omni-MATH and SuperGPQA questions (none from the evaluated rows).
• Dense weights: Q8_0. Engram: unquantized, read from the official shards.

Pipeline: github.com/lxe/dsx/tree/main/quant.

Files

This is build v4b. Earlier builds: revision v4 (rotation without the refit, KL 0.382), v3a (reasoning-trace calibration without rotation, KL 0.397), v2 (agent-text calibration, KL 0.395) and v1 (KL 0.462).

Credits

DeepSeek for the model (MIT) · Xolotl / Strata for the GSQ tools and V4.1 GGUF format (MIT) · DeepSeek for the reference model.py the calibration runs · GPTQ (Frantar et al.) · Hadamard rotation as in QuIP# (Tseng et al.) and QuaRot (Ashkboos et al.) · Lucebox for the eval set · antirez for the calibration prompts · llama.cpp (Georgi Gerganov and the ggml authors, MIT) for the web UI.