DSpark MTP draft head s3recap-p25-b0.1 for DeepSeek-V4-Flash-0731-REAP

This is a fine-tuned DSpark MTP draft head for 0xSero/DeepSeek-V4-Flash-0731-REAP, together with an archive of every other head trained in the same programme. It is not a standalone language model. It replaces the checkpoint's own draft head for speculative decoding. On a Jetson AGX Thor with the author's CUDA engine, the frozen 8-prompt suite decodes at 28.38 tok/s at draft width 5, against 14.61 tok/s plain autoregressive decode in the same run (1.94x).

If you want to download one thing and run it, use DeepSeek-V4-Flash-0731-REAP-DSpark. It is the full checkpoint with this head already installed.

Overview

Which version should I use

Files

Using it

The loadable head is the target checkpoint with these tensors substituted into shards 46–48, re-quantised to the checkpoint's formats by tools/build_trained_head.py in the engine repo. That tool computes fresh power-of-two E8M0 block scales and refuses any tensor whose round-trip error exceeds 0.10.

git clone https://github.com/patrickbdevaney/deepseek-v4-flash-0731-cuda && cd deepseek-v4-flash-0731-cuda
python3 tools/build_trained_head.py \
    --base /path/to/DeepSeek-V4-Flash-0731-REAP \
    --trained /path/to/mtp_trained.safetensors \
    --out /path/to/head_out

The shards installed in DeepSeek-V4-Flash-0731-REAP-DSpark match the SHA-256 values recorded in this repo's head_card.json. The head has only been run with the author's CUDA engine on a Jetson AGX Thor (sm_110a). No other runtime has been tested.

Evaluation

The protocol is the engine's frozen 8-prompt suite, one prompt per category, with at least 200 generated tokens each, greedy decoding and draft width 5. τ is the number of tokens committed per target forward; the maximum is 6 at width 5. Each run checks the first 8 speculative tokens against plain AR (LOSSLESS GATE ... PASS). Under greedy verification, output is the same as plain greedy decoding by construction.

Per prompt (this head):

Compared with the stock head at width 5, τ rises in six categories and falls in two: long_context (5.00 to 4.53) and agentic_format (4.41 to 4.16). Under the project's release rule, the head fails 3 of 6 category floors (long_context, agentic_format, reasoning). It was promoted on the suite mean. The release pointer was not moved to it (HEAD_REGISTRY.md).

The project's end-to-end figure is 22.66 to 28.38 tok/s (+25.3%). The 22.66 is the stock head at width 6 on an earlier engine revision, so that gain combines the head, the width change and engine work.

Archived heads

A head is promoted when its suite τ beats the incumbent by more than 3.5%, the measured run-to-run spread. tok/s values come from different engine revisions and base AR varied between 13.7 and 14.6 tok/s, so compare τ within one width rather than tok/s. Values below come from each arm's head_card.json and the engine's HEAD_REGISTRY.md.

Each arm directory holds mtp_trained.safetensors and head_card.json, plus eval.log and train_metrics.json where they exist. The exceptions: shipped-dspark-0731reap has only a head_card.json, and agentic-p25-b0.1-c3of7 has no logs.

How it was made

• Data. 1,536 prompts were balanced over 8 categories; 1,472 were used for training and 64 held out. Continuations were generated by the target checkpoint itself, and its hidden states were captured over those sequences. The agentic_format prompts came from BFCL.
• Training. One pass of 1,472 steps in three chunks, 16 positions per sequence. Peak learning rate 5e-5, 73 warmup steps, then cosine decay. The cross-entropy and total-variation loss weights were 0.1 and 0.9. Deficit weighting multiplied each block's loss by min((1 − r)/mean(1 − r), 3.0), where r is that block's accepted fraction. A KL anchor with β = 0.1 pulled toward the pre-training head, scaled by r. The confidence-loss weight was 0. Loss went from 1.627 to 0.037 (tail mean 0.739). Sources: train_metrics.json, BEST_SETUP.md.
• β bracket. β = 0 (s3recap-deficit) fell short of the bar. β = 0.5 (s3recap-p25-b0.5) landed below the incumbent.

Limitations

• Measured on one Jetson AGX Thor with one custom engine. No other runtime has been tested.
• The head is specific to this 160-expert REAP checkpoint.
• Trained heads in this programme have lost to the untrained head on held-out continuation drafting against a paired control. The β anchor is meant to limit this. That it does so is not established.
• The four best block-5 arms are within 1.3% τ of each other, inside the 3.5% bar. Picking this head over auto-ace0p5 or auto-ace0p3 is not statistically resolved.
• The three root .md files are 2026-08-25 snapshots of the engine repo. The engine repo is the current reference.

Changelog and corrections

• 2026-08-23: head uploaded; per-category table added, with the note that it does not pass the release rule.
• 2026-08-23 to 2026-08-26: arms archived under arms/. Until 2026-11-06 the README was a copy of the engine repo's development status page, not a model card.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors.

License and attribution

MIT, following the target checkpoint. Upstream model by DeepSeek (deepseek-ai/DeepSeek-V4-Flash-0731, MIT). REAP pruning by 0xSero (0xSero/DeepSeek-V4-Flash-0731-REAP, MIT).

Citation

@misc{devaney2026dsv4s3recap,
  title  = {DSpark MTP draft head s3recap-p25-b0.1 for DeepSeek-V4-Flash-0731-REAP},
  author = {Devaney, Patrick},
  year   = {2026},
  url    = {https://huggingface.co/patrickbdevaney/dspark-mtp-draft-head-s3recap-p25-b0.1}
}