DeepSeek-V4.1-Flash, GGUF with the engram tables
DeepSeek-V4.1-Flash (552B MoE, 40 layers, 384 routed experts, hyper-connections, CSA2 sparse attention, a 189 GiB n-gram conditional memory) converted for llama.cpp without touching the released precision: the routed experts are a lossless repack of the released MXFP4 blocks, attention and dense weights are dequantized from fp8 and stored as Q8_0/BF16, and the two engram tables travel as their raw fp8 bytes with their scales. 502 GB in 11 shards.
It runs only on the dsv41-porte branch of
JigSawPT/llama.cpp. Upstream llama.cpp has no runtime for
V4.1 yet; its open conversion PR (#28696)
stores the engram tables differently, so this file is not interchangeable with files from that
converter. Reconciling the two is part of the plan to upstream the branch.
Text only. DeepSeek-V4.1-Flash also takes images, but the converter on this branch skips the
vision tower: the 11 shards hold 1 012 tensors and none of them is a vision or projector tensor.
Image input is not supported.
If you need image input, a community runtime and GGUF set exist: smalinin/llama.cpp, branch
my_build_deepseek41 with
smalinin/DeepSeek-V4.1-Flash-GGUF (CUDA only,
includes an mmproj file). We have not tested them. Upstream PR
#28696 now carries a full conversion and runtime path for
V4.1 (deepseek41), still unmerged; its files are not interchangeable with these.
What it needs, and what it gives
Measured on one RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM and the file on a PCIe 5 NVMe:
| decode | time to first token | |
|---|---|---|
| new content (our benchmark: 4 prompts of mixed content x 3 rounds) | 5.1 tokens/s | 7.7 s |
| resident content (the same prompt again) | 21.4 tokens/s | 0.26 s |
The experts stream from disk through a VRAM cache (18 GiB) and a pinned host tier (72 GiB); the engram tables are memory-mapped and read 48 rows per token. Nothing here fits in RAM + VRAM and nothing has to. On the buffered read path the ceiling of this architecture on this machine is 6.2 tokens/s without any disk miss; 21 tokens/s needs the working set on the card.
Exactness against the reference implementation: logit correlation 0.9967 at 1 401 tokens, equal to the port against itself across two runs (0.9959). The remaining gap sits at the rounding floor of the reference's own fp8 arithmetic. Details, method and every negative result: the report.
Run
llama-server -m DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 ^
--moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --moe-stream-direct --reasoning off ^
--host 127.0.0.1 --port 8080
All 11 shards go in the same directory; point -m at the first. A bare number after
--moe-stream-cache is a budget in GiB: 18 GiB is about 25 expert slots per layer on this file.
The minimum is 18 slots per layer, --moe-stream-cache 18s, about 13 GiB. --moe-stream-l2 above
72 GiB on a 125.7 GiB machine is
slower: it steals page cache from the engram tables. --moe-stream-io-threads 1 makes runs
bit-for-bit reproducible at 3.6 instead of 4.3 tokens/s. Chat mode (--reasoning off) is the
measured configuration; thinking mode at temperature 0 loops on vague prompts.
--moe-stream-direct reads the experts with O_DIRECT (FILE_FLAG_NO_BUFFERING on Windows) and
falls back to buffered reads where the filesystem refuses it. Added to the line on 23/09/2026: on the
four benchmark prompts it doubles decode on the first pass, when the experts are not cached yet
(3.9–4.9 → 8.9–10.0 tokens/s), and leaves resident decode unchanged (19–22 tokens/s). The table
above was measured without it.
The draft head is published separately (DeepSeek-V4.1-Flash-DSpark-GGUF); on this hardware it is neutral on mixed content.
Files
| shard | GB | contents |
|---|---|---|
-00001-of-00011 |
0.01 | metadata (tokenizer, chat template), engram hashing constants |
-00002-of-00011 |
98.3 | engram table, layer 1 (fp8, raw) |
-00003-of-00011 |
3.1 | engram table scales, layer 1 (raw) |
-00004-of-00011 |
98.3 | engram table, layer 14 (fp8, raw) |
-00005-of-00011 |
46.4 | engram table scales, layer 14; routed experts (MXFP4), layers 0–5 |
-00006 .. -00010 |
45.7 each | routed experts (MXFP4), layers 6–37 |
-00011-of-00011 |
27.1 | routed experts (MXFP4), layers 37–39; attention, shared experts, router, hyper-connections, norms, engram projections, token embedding and output head (Q8_0/BF16/F32) |
SHA256SUMS.txt covers all eleven; run sha256sum -c SHA256SUMS.txt in the download directory.
The 299 GB variant without the engram tables is not
published: zeroing the memory is the exact identity of the module, but the model that comes out
does not produce the released model's outputs.
How it was made
python convert_hf_to_gguf.py <DeepSeek-V4.1-Flash> --outtype bf16 --engram --outfile DeepSeek-V4.1-Flash-MXFP4-engram.gguf
llama-gguf-split --split --split-max-size 48G DeepSeek-V4.1-Flash-MXFP4-engram.gguf DeepSeek-V4.1-Flash-MXFP4-engram
From the released checkpoint, on the same branch. The MXFP4 repack was verified block by block (480/480 identical); the split was verified by loading the first shard and comparing the answer with the monolithic file.
Credits
DeepSeek for the model and the reference implementation (MIT). nibor1896 for Crow, whose expert-streaming patch series the branch builds on (MIT). ggml-org/llama.cpp. Engineering assisted by Claude (Anthropic).