DeepSeek-V4.1-Flash-1017-REAP192-GGUF

This repository holds DeepSeek-V4.1-Flash-1017 pruned to 192 of 384 routed experts per layer, as one GGUF quantization level (L3, 85.6 GiB). It also includes a vision projector (mmproj) and the model's own DSpark drafter. The files run on a llama.cpp branch that reads the 189 GiB Engram tables from NVMe, so they need no memory.

It is aimed at one machine with 128 GB of memory and a local NVMe drive. On a Jetson AGX Thor it decodes at 12.3–12.5 tok/s, and at 17.3–20.7 tok/s with the DSpark drafter, as measured by the author on 2026-10-30.

Release status. L3 failed its pre-registered release gate on top-1 agreement: 0.7725 against a floor of 0.78 (95% CI 0.712–0.829). Its kl32 of 0.508 passes. The owner released it on 2026-10-26 as a near-miss, after its generation smoke test passed. The higher-precision level L2 failed both bars and is not published. The MXFP4 master this was made from was published under a gate amended after its result; see the master card.

Model overview

Files

The splits exist because Hugging Face limits a file to 50 GB. llama.cpp opens the other two splits by itself when given the first. The header sets eos_token_id and eot_token_id to 1 (<｜end▁of▁sentence｜>) and embeds the V4.1 chat template.

Which version should I use

On a Jetson AGX Thor, the Thor build with the CUDA server is about twice as fast and scores slightly higher (top-1 0.7756 against 0.7725, though measured on different token sets). Use this GGUF if you want llama.cpp, or another CUDA GPU with 128 GB of memory (untested).

Quickstart

These steps follow the branch's thor/README.md. They were run on a Jetson AGX Thor (CUDA 13.0). Other GPUs are untested.

# 1. llama.cpp branch: upstream PR #28696 plus the V4.1 / Thor commits
git clone -b deepseek-v4.1-reap-thor https://github.com/patrickbdevaney/llama.cpp && cd llama.cpp
thor/build.sh                      # sm_110 by default; CUDA_ARCH=<sm> for another GPU (untested)

# 2. this repository
hf download patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-GGUF --local-dir ~/models/DeepSeek-V4.1-Flash-1017-REAP192-GGUF

# 3. Engram row files, built once from base shards 47/48 (see below), at $MODEL_DIR/engram_rows or ENGRAM_DIR

# 4. serve (OpenAI-compatible, http://127.0.0.1:8080)
thor/start.sh                      # text + images, 16K context
DSPARK=1 thor/start.sh             # DSpark speculative decoding (text only)

start.sh runs llama-server -ngl 999 -np 1 --jinja. It refuses row files on removable media, and it refuses to start beside another model-loading process. The settings MODEL_DIR, ENGRAM_DIR, CTX, VISION, DSPARK, DRAFT_N, P_MIN, HOST, PORT and EXTRA are environment variables. The direct command is:

LLAMA_ENGRAM_ROWFILE_DIR=/nvme/engram_rows build-thor/bin/llama-server \
  -m ~/models/DeepSeek-V4.1-Flash-1017-REAP192-GGUF/DeepSeek-V4.1-Flash-1017-REAP192-L3-00001-of-00003.gguf \
  --mmproj ~/models/DeepSeek-V4.1-Flash-1017-REAP192-GGUF/mmproj-DeepSeek-V4.1-Flash-1017-bf16.gguf \
  -c 16384 -np 1 -ngl 999 --jinja --port 8080
# DSpark instead of vision: drop --mmproj and add
#   -md .../dspark-DeepSeek-V4.1-Flash-1017-Q8_0.gguf -ngld 999 --spec-type draft-dspark --spec-draft-n-max 3 --spec-draft-p-min 0.5

Engram row files (required, not included)

The two Engram tables (layers 1 and 14) are unchanged by pruning and are not in the GGUF. They are read per token with O_DIRECT from two .v41rows files on a local NVMe drive: 202,758,040,592 bytes (188.8 GiB) together. You build those files once from the base model's shards 47 and 48, using ops/s09_engram_rowfile.py in the pipeline repository. The format is described in llamacpp/ENGRAM_ROWFILE.md. The same files serve the Thor build.

git clone https://github.com/patrickbdevaney/deepseek-v4.1-flash-reap-thor && cd deepseek-v4.1-flash-reap-thor
python3 -m venv .venv && .venv/bin/pip install numpy huggingface_hub
.venv/bin/hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 config.json model.safetensors.index.json \
    model-00047-of-00048.safetensors model-00048-of-00048.safetensors \
    --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --local-dir base     # 2 x 101.5 GB, any disk
export V41_SRC=$PWD/base V41_WORK=$PWD/work
.venv/bin/python ops/s09_engram_rowfile.py build  --dst /nvme/engram_rows
.venv/bin/python ops/s09_engram_rowfile.py verify --dst /nvme/engram_rows --full

Once the row files are built and verified, the base shards can be deleted.

Evaluation

Method. Each published level is scored on its own GGUF, served by llama.cpp (llamacpp/tools/v41-teacher-kld.cpp). The comparison is paired and teacher-forced against the cached outputs of the unpruned model. It uses the 22 text rows of the pre-registered confirmation split: 180,224 tokens at 8,192-token context, all layers on the GPU. No pruning, quantization or release decision looked at this split. Intervals are 95% row (document) bootstrap intervals. The metrics are defined on the master card. Source: s09/L3/teacher_eval/teacher_kld.json and s09/gates.json, copied into gguf_manifest.json.

L2 spends more bits on experts than L3 but scored worse on the same tokens. It was not investigated further and is not published. The ladder script also defines L0 (MXFP4 experts copied losslessly, about 152 GiB) and L1 (about 3.4 bpw). Neither was published.

L3 by domain (point estimates; each domain has 1 to 5 documents):

For reference, on the full 26-document confirmation split (185,565 tokens, vision rows included), the MXFP4 master scores kl32 0.4405 and top-1 0.7980, and the Thor build 0.5194 and 0.7756. Those numbers come from a different harness on a different token set, so compare them with L3 only loosely.

Vision. The mmproj encoder was golden-tested on CPU and CUDA against the reference torch ViT and aligner in f32. Every token row matched at least as closely as the reference's own bf16 path. In one end-to-end spot check, on llama.cpp's tools/mtmd/test-1.jpeg newspaper page, the model read the masthead, the date, the headline and both column headlines correctly. Image-domain quality after pruning was not benchmarked through llama.cpp.

DSpark drafter. All 78 tensors are byte-identical to an independent conversion of the reference drafter, and every reference metadata key is present (s09/dspark2/source_check.json).

Not measured. There are no generative benchmarks, no tool-calling accuracy and no quality past 32K tokens.

Hardware & memory

The numbers below were measured by the author on one Jetson AGX Thor (128 GB unified memory, CUDA 13.0) on 2026-10-30. They are reported in the branch's thor/README.md. The setup was thor/start.sh at its defaults (16K context, all layers on the GPU), with greedy sampling and one stream over /v1/chat/completions.

The release smoke test on 2026-10-26 (ops/logs/s09_smoke_L3.log, 8K context) decoded 52 tokens at 11.86 tok/s. With DSpark, every emitted token is still the target model's greedy pick. The batched verify sums in a different order, though, so output can differ from plain decoding at near-ties.

Context length

Past 16,384 tokens, V4.1 restricts later layers to a two-level candidate-block selection (2048 blocks of 8). The published GGUF predates the metadata keys that switch this on in the branch. So by default the branch clamps the context to 16,384. CTX=32768 thor/start.sh passes the keys with --override-kv. On a 32,767-token held-out row at positions 16,384–32,766, the mask was within noise of running without it: KL +0.00055 [−0.00001, +0.00122]. A 32K-token prompt takes about 14.6 minutes to process. 64K and beyond have not been validated.

How it was made

1. Pruning. The experts were pruned with HOPE (hope_sum_pf0), 192 of 384 per layer, as bit-exact MXFP4 surgery. See the master card for the calibration corpus and the selection.
2. Conversion. llamacpp/convert_v41_reap.py produced a lossless source GGUF from the master: experts as bit-exact GGML MXFP4, FP8 weights as exact BF16, and hyper-connection, norm and sink tensors as F32. Engram tables and MTP layers were left out.
3. Quantization. ops/s09_gguf_ladder.py quantized L3 once, from that source and not from another level, with llama-quantize. It used an importance matrix built from the pipeline's own teacher-pass activations, remapped to the 192-expert keep-set: 240 entries, 552 chunks, held-out rows excluded. The keep-set fingerprint is checked at conversion, at imatrix load and on the written splits.
4. Branch. The branch is upstream PR #28696 (head 2076412598e9) plus commits for: F32 hyper-connections; reader layers reusing their index source's top-k; the Engram O_DIRECT row store; the REAP heal-gain hook (unused here); DSpark drafter export; the 16K clamp; the candidate-block mask; a faster indexer transpose; and the deepseek41v vision projector. The release was built at 92031e6a; the branch head is cf4d51ba2.

Limitations

• The release gate failed as registered (top-1 0.7725 < 0.78). This level was released as a near-miss by owner decision.
• Multilingual quality is much lower than in the unpruned model (top-1 0.50 on L3). For non-English use, prefer the unpruned model.
• Context is 16K by default and 32K validated. Long prompts are slow on this branch.
• Tool calling. The calibration corpus carried no tool-call turns, and tool-call formatting is not evaluated.
• Vision was spot-checked, not benchmarked. A low-resolution 531×229 crop made the model loop. DSpark was measured on text only.
• Only measured on a Jetson AGX Thor. Partial GPU offload and other GPUs are untested.

Changelog / corrections

• 2026-10-26: L3 published (s12 card generated from s09/s10 artifacts).
• 2026-10-30: vision mmproj and DSpark drafter added. Measured Thor serving numbers and the 32K context validation added.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors. Corrections: the published ladder is only L3. L0, L1 and L2 are not in this repository, and L2 failed its gate. The memory breakdown and the "87.30 GiB resident" figure in the 2026-10-26 rewrite could not be traced to any log and were removed. The Engram row-file commands now match the script (--dst, with V41_SRC); the rewrite's --src/--out flags do not exist. The calibration long-context rows are 32K–64K tokens, not 32K–128K.

License & attribution

MIT, the same as the base model. The weights are derived from DeepSeek-V4.1-Flash-1017 by DeepSeek-AI. The runtime builds on llama.cpp and its PR #28696 (deepseek41 architecture).

Citation

@misc{devaney2026dsv41reap192gguf,
  title        = {DeepSeek-V4.1-Flash-1017-REAP192-GGUF: 192-of-384 expert-pruned DeepSeek-V4.1-Flash-1017 for llama.cpp with Engram on NVMe},
  author       = {Patrick Devaney},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-GGUF}}
}