DeepSeek-V4.1-Flash, GGUF with the engram tables
DeepSeek-V4.1-Flash (552B MoE, 40 layers, 384 routed experts, hyper-connections, CSA2 sparse attention, a 189 GiB n-gram conditional memory) converted for llama.cpp without touching the released precision: the routed experts are a lossless repack of the released MXFP4 blocks, attention and dense weights are dequantized from fp8 and stored as Q8_0/BF16, and the two engram tables travel as their raw fp8 bytes with their scales. 502 GB in 11 shards.
It runs only on the dsv41-porte branch of
JigSawPT/llama.cpp. Upstream llama.cpp has no runtime for
V4.1 yet; its open conversion PR (#28696)
stores the engram tables differently, so this file is not interchangeable with files from that
converter. Reconciling the two is part of the plan to upstream the branch.
What it needs, and what it gives
Measured on one RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM and the file on a PCIe 5 NVMe:
| decode | time to first token | |
|---|---|---|
| new content (our benchmark: 4 prompts of mixed content x 3 rounds) | 5.1 tokens/s | 7.7 s |
| resident content (the same prompt again) | 21.4 tokens/s | 0.26 s |
The experts stream from disk through a VRAM cache (18 GiB) and a pinned host tier (72 GiB); the engram tables are memory-mapped and read 48 rows per token. Nothing here fits in RAM + VRAM and nothing has to. The ceiling of this architecture on this machine is 6.2 tokens/s without any disk miss; 21 tokens/s needs the working set on the card.
Exactness against the reference implementation: logit correlation 0.9967 at 1 401 tokens, equal to the port against itself across two runs (0.9959). The remaining gap sits at the rounding floor of the reference's own fp8 arithmetic. Details, method and every negative result: the report.
Run
llama-server -m DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 ^
--moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --reasoning off ^
--host 127.0.0.1 --port 8080
All 11 shards go in the same directory; point -m at the first. --moe-stream-cache has a
minimum of 18 slots per layer (13 GiB). --moe-stream-l2 above 72 GiB on a 125.7 GiB machine is
slower: it steals page cache from the engram tables. --moe-stream-io-threads 1 makes runs
bit-for-bit reproducible at 3.6 instead of 4.3 tokens/s. Chat mode (--reasoning off) is the
measured configuration; thinking mode at temperature 0 loops on vague prompts.
The draft head is published separately (DeepSeek-V4.1-Flash-DSpark-GGUF); on this hardware it is neutral on mixed content.
Files
| shard | GB | contents |
|---|---|---|
-00001-of-00011 |
0.01 | metadata, small tensors |
-00002-of-00011 |
98.3 | engram table, layer 3 (fp8, raw) |
-00003-of-00011 |
3.1 | |
-00004-of-00011 |
98.3 | engram table, layer 13 (fp8, raw) |
-00005 .. -00010 |
45.7–46.4 each | routed experts (MXFP4), attention, norms |
-00011-of-00011 |
27.1 |
SHA256SUMS.txt covers all eleven. The 299 GB variant without the engram tables is not
published: zeroing the memory is the exact identity of the module, but the model that comes out
does not produce the released model's outputs.
How it was made
python convert_hf_to_gguf.py <DeepSeek-V4.1-Flash> --outtype bf16 --engram --outfile DeepSeek-V4.1-Flash-MXFP4-engram.gguf
llama-gguf-split --split --split-max-size 48G DeepSeek-V4.1-Flash-MXFP4-engram.gguf DeepSeek-V4.1-Flash-MXFP4-engram
From the released checkpoint, on the same branch. The MXFP4 repack was verified block by block (480/480 identical); the split was verified by loading the first shard and comparing the answer with the monolithic file.
Credits
DeepSeek for the model and the reference implementation (MIT). nibor1896 for Crow, whose expert-streaming patch series the branch builds on (MIT). ggml-org/llama.cpp. Engineering assisted by Claude (Anthropic).