DeepSeek-V4.1-Flash-Spark

DeepSeek-V4.1-Flash with every backbone routed expert re-encoded as EXL3 trellis (MUL1 codebook) at a per-expert K2 / K3 / K5 mix, sized to serve on two NVIDIA DGX Spark (GB10, 2 x 128 GB unified memory, tensor parallel 2). Average routed-expert rate 2.77 bpw. Everything else — attention, shared experts, routers, Engram tables, the DSpark/MTP draft layers and their experts, the vision tower, embeddings and head — is the upstream checkpoint, byte for byte, in its native FP8 / BF16 / MXFP4 formats.

[!IMPORTANT] EXL3 experts need the EXL3 MoE path in the companion vLLM image (below). Stock vLLM and plain transformers cannot run this checkpoint. Two Engram shards are stored as split parts (HF limits files to 50 GB); run ./reassemble.sh once after download.

At a glance

Quality

Offline full-vocabulary token-wise KL divergence against the native checkpoint (teacher), 64 windows per panel, bootstrap 95 % CI. Lower KLD and higher top-1 agreement are better.

Paired against the previous allocation at 2.68 bpw (same K2/K3 banks, 1,782 experts left as native MXFP4), this build moves −1.5 % on v3.1 and +0.8 % on legacy KLD, both inside the noise band; it was accepted because every routed expert now uses one kernel family.

Serving on 2 x DGX Spark

Historical measurements on host-built image sha256:5668e35e5ee021da4b9ce88a1964caf0e082e8ca01d81d1d64b6213a55c6add5 and the full upstream checkpoint layout, TP2 over the Sparks' RoCE link, FP8 KV cache, DSpark speculative decoding (7 draft tokens), Engram tables on disk, CUDA graphs FULL_DECODE_ONLY, 2 concurrent sequences:

All decode runs used the model's default sampling and stopped naturally (3k-15k tokens); no output caps. Smoke checks passed for text, tool calls and vision.

Published image and validation status

The public CI image is ghcr.io/0xsero/deepseek-v4.1-flash-spark@sha256:3cbc8ec016f5fbfc82eba3480de12399cbce31e7b76aa3108b8fe8246c42a79d. GitHub-hosted ARM64 build 37138684406 published it from source 93e001cf4ecafeb71c6e737eeefad9a9b4ce9235 on main, with verified signed provenance. Fresh model acceptance on this exact image and this repository's compact layout is still pending in registry PR #154. The historical measurements above have not been transferred to this image or layout.

Run it

1. Download and rebuild the two Engram shards:

hf download 0xSero/DeepSeek-V4.1-Flash-Spark --revision 08ac8b3defc9a239ba0baf51059687c689406519 --local-dir ~/models/DeepSeek-V4.1-Flash-Spark
cd ~/models/DeepSeek-V4.1-Flash-Spark && ./reassemble.sh     # cat parts, sha256 check

Allow about 665 GB per node during download and reassembly, plus image and cache space.

2. The two-Spark launcher mounts the repository root at /model and writes a separate plan copy whose bank paths are absolute /model/exl3/... paths. It mounts that copy at /plans and keeps the checkpoint read-only. The pending registry recipe instead uses the original relative plan with workdir /model/exl3.

3. To exercise the CI image, set IMG to the exact public digest above when running scripts/launch.sh from the launcher repository; also set WORKER to your second Spark's SSH target as its instructions describe. The launcher's default still points to the historical host image. This is a candidate configuration pending the acceptance run linked above. Key flags:

vllm serve /model --tensor-parallel-size 2 --nnodes 2 --dtype bfloat16 \
  --kv-cache-dtype fp8 --block-size 256 --swa-block-size 128 --kv-cache-memory-bytes 2700000000 \
  --max-model-len 262144 --max-num-seqs 2 --max-num-batched-tokens 2048 \
  --attention-backend B12X --linear-backend b12x --moe-backend b12x \
  --engram-config '{"table_memory":"disk","cpu_offload":false}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_tensor_parallel_size":2}' \
  --tokenizer-mode deepseek_v41 --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice
# launcher env: ST_EXL3_PLAN=/plans/J268-ho-v31-K5n.json (absolute qdir), ST_EXL3_PREFILL=st

The validated runs mounted the full upstream checkpoint as /model; this repo carries the same tensors minus the backbone routed experts, which the EXL3 path never reads.

Files

Verification

• Dropped exactly the 92,160 backbone routed-expert tensors (layers.{0-39}.ffn.experts.{0-383}.w{1,2,3}.{weight,scale} = 40 x 384 x 6). Nothing else was dropped; the mtp.* draft experts are kept.
• Every kept tensor's bytes were hashed at read time and re-hashed from the written shard: all 3,913 non-Engram tensors match, dtypes and shapes match. Engram shards are verified by whole-file sha256 against the source. Receipt: repack-report.json.

Provenance

License and credits

MIT, carried from DeepSeek's release; this repo adds no terms. Thanks to DeepSeek for the model, turboderp for ExLlamaV3 / EXL3 (MIT) — the trellis encoder and kernels used here — and Local Inference Lab and the vLLM community for the Spark serving stack.