DeepSeek-V4.1-Flash-1017 · dsx MXFP4 + 3-bit

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) in 270 GB, built for one AMD Instinct MI325X with exact routing. The experts and dense weights run from HBM; the separate Engram table uses NVMe and the host page cache. Most routed experts keep the checkpoint's own MXFP4 weights; the least-used ones get a fitted 3-bit codebook. Runs on dsx's AMD runner (amd/, Rust + HIP). For a desktop with NVIDIA GPUs, see dsx-t16.

Which build

All four run on dsx and were measured on the same eval (Lucebox 1017, 8 x 1,024 tokens against the FP8 model's top-256 log-probs).

The table retains the original model-release measurements; quality was measured without expert deferral. Decode speeds were not all measured with the same runtime revision. For the later NVIDIA kernel integration, current switches and the unresolved decode exactness check, see measurement status.

At a glance

Quality

Measured on Lucebox's 1017 eval set: 8 × 1024 tokens against the FP8 model's top-256 log-probs, teacher-forced through the decoder.

Sizes exclude the Engram tables (and, for the original, the vision and draft layers). Measured on these shards; another run gave KL 0.047: float atomics in the runner sum in varying order, which flips a few near-tied expert choices.

Greedy reasoning (temperature 0, thinking on, reasoning effort low), two competition-math prompts: 6,624 and 13,482 generated tokens with boxed answers. The hosted model needed 5.6K and 21K on the same prompts.

Speed

One MI325X (256 GiB HBM3E), greedy, exact routing.

Long prompts slow down with length because the attention indexer scores every earlier compressed position.

Serving profiles

The same six GGUF shards support both profiles. At load time, the runner can re-quantize additional cold MXFP4 experts to 3-bit to leave room for drafts and concurrent sessions. No new GGUF download or export is needed.

The coding profile also measured 246 decode tokens/s in a single-session code replay and about 61 per session with eight sessions (486 total). Measurements used temperature 0.1, top_p 1.0 and thinking off. They are short all-active decode windows, excluding prefill; earlier four-session trials reached as low as 88 tokens/s per session. The quality figures for this profile come from an earlier evaluation with the same expert placement, not a fresh evaluation of the final serving build. The published-build quality and speed tables above retain their original configuration.

The coding profile target-verifies the checkpoint's native five-token MTP proposals, supports sampled requests, and favors decoding between prompt chunks. See the AMD README and profile settings for memory and scheduling details.

Run

# the quant: GGUF shards, loaded as one model
hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-mxfp4 --local-dir dsx-mxfp4

# config, tokenizer and Engram tables from the official checkpoint
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 \
  --include config.json tokenizer.json model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors
cp dsx-mxfp4/dsx-engram.json v41-1017/

# dsx's AMD runner: ROCm 7 (gfx942), Rust
git clone https://github.com/lxe/dsx && cd dsx && cargo build --release -p dsx-amd

DSXA_WEIGHTS=../dsx-mxfp4/DeepSeek-V4.1-Flash-1017-dsx-mxfp4-00001-of-00006.gguf \
  ./target/release/dsx-amd serve --model ../v41-1017 --ctx 262144 --host 0.0.0.0 --port 8090

Open http://localhost:8090/ for the web UI, or point any OpenAI-compatible client at http://localhost:8090/v1.

Coding profile

After the download and build above, run these commands from the dsx repository. The native draft stack also needs the official checkpoint's shards 44–46:

hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir ../v41-1017 \
  --include model-00044-of-00048.safetensors model-00045-of-00048.safetensors model-00046-of-00048.safetensors

DSX_MODEL_DIR="$PWD/../v41-1017" \
DSXA_WEIGHTS="$PWD/../dsx-mxfp4/DeepSeek-V4.1-Flash-1017-dsx-mxfp4-00001-of-00006.gguf" \
DSX_RANK="$PWD/../dsx-mxfp4/expert-rank.json" \
  bash amd/box/serve-agentic-mi325x.sh

This starts up to eight sessions with a 256K batch context budget, temperature 0.1 and top_p 1.0. API and admin services bind to loopback; use an SSH tunnel for a remote GPU. Set "reasoning_effort": "none" in OpenAI requests to match the thinking-off measurements. Budget additional disk space for the official MTP shards.

Hardware

Other AMD GPUs and smaller memory are not supported by this build. For NVIDIA desktops, use dsx-t16.

How it was made

• Dense weights (attention, shared experts, routers, Engram projections): the checkpoint's FP8 with its E8M0 32×32 block scales, unchanged. Embeddings and the output head: bf16, unchanged.
• Routed experts: the 8,992 most used experts (by routing frequency measured on agentic text) keep the checkpoint's MXFP4 weights. The other 6,368 are re-quantized from those FP4 values to a 3-bit codebook of 8 levels (fitted by Lloyd iterations on a sample of experts, stored as FP8 values) with one e4m3 scale per 32 weights and a multiplier per matrix, round-to-nearest with a per-group scale search.
• The split is chosen so the whole model fits in 256 GiB of HBM next to the caches for 256K tokens of context.

Files

Credits

DeepSeek for the model (MIT) · Lucebox for the eval set · llama.cpp (Georgi Gerganov and the ggml authors, MIT) for the web UI.