DeepSeek-V4.1-Flash-1017-REAP192-Thor

This is DeepSeek-V4.1-Flash-1017 pruned to 192 of 384 routed experts per layer and requantized to 2.83-bit K-quant experts. It is stored in the thor-v1 layout for one 128 GB NVIDIA Jetson AGX Thor. The Engram tables stay on NVMe and are read per token, so the 106.9 GiB build fits in unified memory.

It is served by a dedicated pure-CUDA server, deepseek-v4.1-flash-cuda-server. The author measured it on 2026-10-30 at 25.3 tok/s plain decode and 25.1–40.2 tok/s with the model's own DSpark speculation.

Release status. This build fails its release gate as pre-registered on two clauses. Its top-1 agreement is 0.7756, against a floor of 0.78. It also does not beat a random keep-set on kl32. It was published on 2026-10-26 under two owner amendments made on 2026-10-24, after the result: multilingual was left out of the beat-random test, and the 0.78 top-1 floor was waived for this build. The CUDA layer-by-layer correctness gate also failed as registered (89.1% of disagreements were near-ties, against the 90% rule), and the owner waived it. See Release gates.

Model overview

Files

The layout is described in cuda/FORMAT_PROPOSAL.md. No other runtime reads it.

Which version should I use

On a Jetson AGX Thor this build is the recommended one. On other hardware, use the GGUF.

Quickstart

These steps follow the server's README. They were run on a Jetson AGX Thor with CUDA 13.0. The server targets sm_110a only.

# 1. this repository (weights, NVFP4 overlay, tokenizer)
hf download patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-Thor --local-dir ~/models/DeepSeek-V4.1-Flash-1017-REAP192-Thor

# 2. Engram row files on local NVMe (see below), at $CKPT/engram_rows or ENGRAM_DIR

# 3. build and start (CUDA 13, sm_110a, liburing)
git clone https://github.com/patrickbdevaney/deepseek-v4.1-flash-cuda-server && cd deepseek-v4.1-flash-cuda-server
scripts/build_v41.sh v41_server
./start.sh               # http://127.0.0.1:8000, 32K context, NVFP4 overlay on, DSpark K=3 on

The server provides an OpenAI-style /v1/chat/completions endpoint (streaming, tool calls in the model's DSML encoding, thinking blocks, base64 image parts), /v1/completions, /v1/models, /health, /metrics and a web UI. The settings OVERLAY=0 (FP8 dense instead of NVFP4), SPEC=0 (no speculation) and SEQMAX are environment variables. start.sh refuses to start beside another model-loading process, because on Thor's unified memory two such processes can hard-reboot the box.

Engram row files (required, not included)

The two Engram tables are not pruned and are not in this repository. They are read per token with O_DIRECT from two .v41rows files: 202,758,040,592 bytes (188.8 GiB) together, which must be on a local NVMe drive. You build them once from the base model's shards 47 and 48 with ops/s09_engram_rowfile.py (build --dst DIR, then verify --dst DIR --full, with V41_SRC pointing at a directory holding the base config.json, model.safetensors.index.json and the two shards). The format is described in llamacpp/ENGRAM_ROWFILE.md. The GGUF card has the full command sequence. The same files serve both runtimes.

Evaluation

Method. This is a paired, teacher-forced comparison against the unpruned base model on the confirmation split: 185,565 tokens from 26 held-out documents, never used for any selection or tuning decision. Intervals are 95% document-cluster bootstrap intervals. The metric definitions are on the master card. Source: s10/final/results.json of the build pipeline.

Paired costs:

• Requantizing the experts (thorA vs mxfp4_A) costs kl32 +0.0789 [+0.0571, +0.1024] (gate limit 0.25) and 2.2 points of top-1.
• The NVFP4 overlay (thorA_overlay vs thorA) costs kl32 +0.0182 [+0.0137, +0.0233] (gate limit: upper bound ≤ 0.03) and 0.21 points of confident flips (limit 0.5). It passed its pre-registered gate.
• At similar expert bytes, 192 requantized experts beat 128 bit-exact ones: kl32 −0.2831 [−0.3479, −0.2245], top-1 +6.6 points.

By domain. Each domain has only 1 to 5 documents; math is a single document.

What these numbers are, and are not. thorA and thorA_overlay are this build's weights, scored by the pipeline's teacher-forced replay harness (ops/s10_eval.py), not by the CUDA server. The server was instead checked layer by layer against a torch reference (below). There are no generative benchmarks, no long-context quality past 8K tokens, no tool-calling accuracy and no image-task benchmarks.

Release gates

In the CUDA golden runs, every hidden-state layer check, the agreement rule and the final-cosine rule passed. The gates were registered 2026-10-22 (sha256 169c0bba…); after both amendments the file's sha256 is f95b2672….

Hardware & memory

Everything was run on one NVIDIA Jetson AGX Thor (128 GB unified memory, CUDA 13.0, Linux 6.8.12-tegra).

Fit (estimated). The build occupies 106.9 GiB. The planning figures add 2.0 GiB for the main 1M-token context state (not measured) and 6.0 GiB of workspace, for 114.9 GiB of a 117.0 GiB budget. The Engram tables take no memory.

Release gate measurements (s08, 2026-10-25, the first working engine, FP8 dense, no speculation):

Current server (measured by the author 2026-10-30, ./start.sh defaults: 32K context, NVFP4 overlay, greedy, one stream, as reported in the server README):

The server README also reports these results:

• Speculation averages 1.345× over plain decoding on 27 held-out prompts. Greedy output was token-identical, with 0 divergences.
• Plain decode is about 74% of a 247 GB/s measured bandwidth roofline, at about 7.6 GB read per token.
• Image input through the CUDA vision tower was added on 2026-10-31.

These are the author's runs, not this card's audit; the pipeline's docs/zenith/ holds the history.

How it was made

1. Keep-set. hope_sum_pf0, 192 of 384 experts per layer, chosen on a held-out screen split. It is the same keep-set as the MXFP4 master, which documents the calibration corpus and the selection.
2. Requantization (ops/s07_quant.py). Each kept expert was quantized once, from the MXFP4 source, into ggml Q2_K/Q3_K blocks with GPTQ error-feedback rounding. The per-layer type mix favours the protected layers: the first three layers, the Engram, KV-source and DSpark-tap layers. The heal gain was rejected, so no gain is folded in.
3. Dense path. Copied from the base checkpoint unchanged. The NVFP4 overlay is a separate, optional copy of the dense attention and shared-expert weights.
4. Arm choice. The pre-registered A/B decision at matched expert bytes picked 192 requantized experts over 128 bit-exact ones (paired kl32, CI excludes zero). A model confidence set at α = 0.1 kept only thorA (FP8 dense). The overlay arm is measurably worse and ships because its pre-registered speed-lever gate passed.

Limitations

• The release gate failed as registered (see above). The default served configuration (NVFP4 overlay on) scores top-1 0.7681.
• Multilingual quality is much lower than in the unpruned model (top-1 0.49). For non-English use, prefer the unpruned model.
• Tool calling. The calibration renderer dropped tool-call turns. The release server smoke test returned a malformed tool-call argument string. Tool-call accuracy is not evaluated.
• Long context quality is untested past 8K tokens. Allocating a 1M-token context and serving a 64K prompt worked in the release gate.
• Runtime lock-in. The thor-v1 layout is read only by the CUDA server above, which targets Jetson AGX Thor (sm_110a).
• This repository has no config.json; the architecture lives in thor.json. The base model's MIT LICENSE was added on 2026-11-06.

Changelog / corrections

• 2026-10-26: published (s12 card generated from s10/s08 artifacts).
• 2026-10-30: tokenizer files and the NVFP4 overlay added. Measured CUDA-server numbers added.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors. Corrections: there is no dense.safetensors; the dense path lives in layer-NN.safetensors and globals.safetensors. There is no config.json in this repository (the base model's MIT LICENSE was added the same day). The MXFP4 master is 151.6 GiB, not 146.4 GiB. The overlay is 3.42 GiB, not 3.5. Speculation is on by default in the server; an earlier section said it was off. The card no longer says "text only": the server accepts images since 2026-10-31. The heal gain is 1.0, so nothing was folded into w2. The keep-set used no per-domain protection (protect fraction 0). The default served configuration (thorA_overlay) is now reported beside thorA.

License & attribution

MIT, the same as the base model. The weights are derived from DeepSeek-V4.1-Flash-1017 by DeepSeek-AI. Its licence is copied unchanged in LICENSE.

Citation

@misc{devaney2026dsv41reap192thor,
  title        = {DeepSeek-V4.1-Flash-1017-REAP192-Thor: a 192-of-384 expert-pruned DeepSeek-V4.1-Flash-1017 for one Jetson AGX Thor},
  author       = {Patrick Devaney},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-Thor}}
}