DeepSeek-V4.1-Flash-1017 TrellisMX K2

A 2-bit TrellisMX build of deepseek-ai/DeepSeek-V4.1-Flash-1017 (revision 1bae0e1088993b9b82de08f59de5be5a34e42170), laid out for two 96 GB RTX PRO 6000 Blackwell cards at tensor parallel 2, with the two Engram tables at 4 bits (about 104 GB) meant to stay in host RAM.

The encode and every measurement on this card ran on an 8x NVIDIA B200 node generously donated by Lambda (@LambdaAPI). Thank you, Lambda.

Built with the SM120 vLLM image by Local Inference Lab and Luke Alonso, the b12x kernels by Luke Alonso with Martin Vit and other contributors, and Local Inference Lab's CSF scale format; TrellisMX was inspired by QTIP and EXL3, and its FP8 trellis decode by Luke Alonso's KQuant/QSRT. Full credits: Credits and CREDITS.md.

Released under the MIT License, the same license as the base model (LICENSE).

Status: research build. The weights and the reference-path measurements below are complete. The serving image verdictai/deepseek-v41-flash-trellismx:k2-fp8-tp2-20261019 runs this checkpoint at tensor parallel 2 on two RTX PRO 6000 Blackwell 96 GB cards with DSpark speculative decoding and full CUDA graphs; it was booted and benchmarked on that hardware on 2026-10-19 (Speed, below). The tag was rebuilt that day with fixes to the startup path and to the expert kernel's token routing: if you pulled it earlier on 2026-10-19, pull it again. serving/ holds its docker-compose.yml, serve.sh and SERVING.md.

2026-10-19: 43 of the 48 model-000NN-of-00048.safetensors shards were rewritten with contiguous tensor offsets. The first upload padded some tensors to 8-byte boundaries, which the safetensors loader refuses. Every tensor's bytes are unchanged; SHA256SUMS lists the new files. If you downloaded this repository before that date, download it again.

Run it

You need two RTX PRO 6000 Blackwell 96 GB cards (SM120) with PCIe peer-to-peer between them, about 100 GiB of free host RAM for the Engram tables, an NVIDIA driver with CUDA 13.4 support, the NVIDIA Container Toolkit, docker compose v2 and about 262 GB of disk.

docker pull verdictai/deepseek-v41-flash-trellismx:k2-fp8-tp2-20261019
hf download brandonmusic/DeepSeek-V4.1-Flash-1017-TrellisMX-K2 --local-dir /models/DeepSeek-V4.1-Flash-1017-TrellisMX-K2
cd /models/DeepSeek-V4.1-Flash-1017-TrellisMX-K2/serving
export CHECKPOINT_ROOT=/models/DeepSeek-V4.1-Flash-1017-TrellisMX-K2
bash serve.sh

serve.sh reads the checkpoint's manifest, picks tensor parallel 2 and GPUs 0,1 (DS41_GPUS changes them), checks the free host RAM and starts the container. The API is OpenAI-compatible on port 8010, model name deepseek-v41-flash-trellismx-k2:

curl -s http://127.0.0.1:8010/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v41-flash-trellismx-k2","messages":[{"role":"user","content":"Hello"}],"max_tokens":256}'

serving/SERVING.md has every setting and its default.

Speed on two RTX PRO 6000 Blackwell cards

Measured on 2026-10-19 with Local Inference Lab's llm-inference-bench v0.4.29 (https://github.com/local-inference-lab/llm-inference-bench) on two RTX PRO 6000 Blackwell Workstation Edition cards (power limits 300 W and 350 W) at tensor parallel 2, with the serving image's defaults: full CUDA graphs, DSpark speculative decoding with 7 draft tokens, FP8 KV cache. 30 seconds per decode cell.

Aggregate decode throughput, tokens per second, by concurrent requests:

DSpark accepted 1.7 to 2.3 tokens per step on the bench's text.

The k2 image was updated later on 2026-10-19 with a faster routed-expert kernel for this checkpoint's shape (the same arithmetic, with each expert's weights decoded once per call instead of once per routed token). In a same-session check on the same two cards it raised single-request decode from 121.5 to 147.4 tokens per second and two-request decode from 164.8 to 221.9 at an empty context; the table above was measured with the earlier kernel, and prefill is unchanged.

Prefill, one request:

The bench's built-in reasoning checks, three runs each at concurrency 3 with no token cap: lavd-test 3 of 3 exact (about 20,200 tokens per answer), estonia 3 of 3 pass, hotel-lights 3 of 3 exact.

KV cache at the default GPU memory utilization of 0.95: 1.29 GiB per card, 1,248,932 tokens (1.19 requests at the full 1,048,576-token context). The first boot also prints the KV_CACHE_MEMORY value that fully uses the cards, about 4.4 GiB per card; serving/SERVING.md explains how to pin it.

What is quantized

Fidelity

KL(official checkpoint || this build) in nats, over the full vocabulary (129,280 ids), in FP64, at every next-token row (row r predicts token r+1). Teacher: the official checkpoint run in DeepSeek's reference implementation, FP32 logits, published in brandonmusic/DeepSeek-V4.1-Flash-1017-Teacher-Logits (revision 68379d81), folders cuts-2048/ and windows-8192/. Intervals are window-block bootstrap (BCa 95%) and Wilson for top-1.

Headline, 2,048-token sequences: the 128 evaluation windows of 8,192 tokens cut into 512 sequences of 2,048 tokens, each run as a fresh sequence from position 0 (2,047 rows each, 1,048,064 rows):

Where the loss comes from (serving formats, same 512 sequences): the official experts with the K4 Engram tables give 0.0454 [0.0407, 0.0511] (top-1 94.10%); the K2 experts with the official Engram tables give 0.1429 [0.1339, 0.1536] (top-1 86.82%). Their sum, 0.1883, exceeds the combined build's 0.1439: the two errors overlap rather than add.

Full 8,192-token windows (8,191 rows each, 1,048,448 rows):

By domain (serving formats, 2,048-token sequences): agentic 0.341 [0.310, 0.375], SWE agent 0.173 [0.151, 0.210], math 0.075 [0.069, 0.083], science 0.098 [0.088, 0.109], competitive programming 0.107 [0.092, 0.130], instruction following and chat 0.091 [0.085, 0.097], multilingual 0.099 [0.094, 0.106], long-context transduction 0.020 [0.017, 0.025], safety 0.150 [0.139, 0.168], web text 0.189 [0.179, 0.206], legal 0.166 [0.149, 0.191]. Unlike the 3-bit builds, KLD rises at 8,192 tokens: the median falls (0.0132 against 0.0201) but the tail grows (p99 3.70 against 2.04).

Checks: the teacher scored against itself is exactly 0 on every window; the reference model's canary window, run at the start and end of every session, is bit-identical; on every window the teacher's top-1 agrees with the realized token inside the [0.20, 0.995] band; the GPU scoring kernel matches a CPU FP64 reference to within 3e-12 on sampled rows of every window. The evaluation windows share at most 5% of their content 13-grams with the data the build was calibrated on, and every accepted conversation shared at most 2% with it.

How it was measured: the base model's layers run in DeepSeek's reference implementation, layer by layer on 4x NVIDIA B200 at model parallel 4, one window per forward, with this checkpoint's experts and Engram rows reproduced bit for bit in that path (the Engram rows are gathered on the CPU, as the serving path does from host RAM). The native SM120 kernel path has not been measured. Every figure with its interval: evaluation/report.json; per-sequence and per-window statistics: evaluation/summary-A1.json .. summary-A4.json; per-token records: evaluation/records/; window sources and token-id SHA-256: evaluation/eval-windows.json.

Files

• model-000NN-of-00048.safetensors + model.safetensors.index.json: every tensor that is not a routed expert or an Engram table, byte-identical to the base release.
• trellismx/tp2/layer-LLL-rank-R.safetensors: the 2-bit routed experts of layer LLL for tensor-parallel rank R (each rank holds all 384 experts' 1,152-wide half of the intermediate dimension), their CSF-compressed UE8M0 scales and the per-layer vectors the runtime applies with the experts.
• trellismx/manifest.json: the format description and the SHA-256 and size of every expert file.
• engram/layer-001.*, engram/layer-014.*: the 4-bit Engram tables (.trellis.safetensors, U8 [rows, 128]) and their UE8M0 scales (.scale.safetensors, F8_E8M0 [rows, 8]).
• config.json (the base config with quantization_config set for this format), tokenizer and encoding/ files from the base release, LICENSE, SHA256SUMS (every file).
• evaluation/: report.json (every figure above with its interval), summary-A1.json .. summary-A4.json (per sequence and per window), records/A1 .. records/A4 (per-token records) and eval-windows.json (every window's sources, token-id SHA-256 and overlap screen). A1 = serving formats, A2 = weights only, A3 = official experts with the K4 Engram tables, A4 = K2 experts with the official Engram tables.
• serving/: docker-compose.yml, serve.sh and SERVING.md for the serving image.

Teacher logits

The FP32 teacher logits used for the KLD figures are in the dataset brandonmusic/DeepSeek-V4.1-Flash-1017-Teacher-Logits (revision 68379d81): folder cuts-2048/ for the 2,048-token sequences and windows-8192/ for the 8,192-token windows, with every window's sources and token-id SHA-256.

Credits

• Compute: Lambda. The encode and every measurement on this card ran on an 8x NVIDIA B200 node that Lambda donated for this work. Our sincere thanks to Lambda (@LambdaAPI, https://x.com/LambdaAPI) for supporting independent open-model research: this build would not exist without that compute.
• Base model. DeepSeek-V4.1-Flash-1017 by DeepSeek (deepseek-ai/DeepSeek-V4.1-Flash-1017), MIT License (LICENSE).
• Runtime: Local Inference Lab and Luke Alonso. The SM120 runtime this checkpoint targets is built on the vLLM image by Local Inference Lab and Luke Alonso (ghcr.io/local-inference-lab/vllm, https://github.com/local-inference-lab), a fork of vLLM (Apache-2.0, vLLM contributors), and the open-source projects that image bundles, each under its own license: FlashInfer, LMCache, InstantTensor, xgrammar and fastokens.
• Kernels: b12x by Luke Alonso. The MoE, attention and GEMM kernels of that runtime come from b12x (https://github.com/lukealonso/b12x, mirrored at https://github.com/local-inference-lab/b12x) by Luke Alonso, with Martin Vit and other contributors (Apache-2.0).
• CSF encoder: Local Inference Lab. The lossless compressed scale format (CSF) the expert scales are stored in, and its encoder and decoder, are Local Inference Lab's work, shipped in Local Inference Lab's FlashInfer.
• TrellisMX: QTIP and EXL3. TrellisMX, Brandon M. Music's trellis-coded format, was inspired by QTIP (Albert Tseng, Qingyao Sun, David Hou and Christopher De Sa, "QTIP: Quantization with Trellises and Incoherence Processing", NeurIPS 2024) and by ExLlamaV3 and its EXL3 format by turboderp and contributors (https://github.com/turboderp-org/exllamav3, MIT License).
• Codec changes: QSRT and Luke Alonso. Decoding the trellis directly into FP8 (E4M3) tensor-core operands was inspired by Luke Alonso's KQuant/QSRT work and the b12x W4A8 trellis path, which decodes the QSRT trellis to E4M3.
• Methods. The encoder's Hessian-based error feedback follows GPTQ (Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, ICLR 2023). The incoherence processing and rotations follow QuIP (Jerry Chee, Yaohui Cai, Volodymyr Kuleshov and Christopher De Sa, NeurIPS 2023), QuIP# (Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov and Christopher De Sa, ICML 2024), QuaRot (Saleh Ashkboos et al., NeurIPS 2024) and SpinQuant (Zechun Liu et al., 2024). The UE8M0 block scales follow the OCP Microscaling (MX) Formats Specification (Bita Darvish Rouhani et al., 2023).
• Evaluation data. The evaluation windows behind the KLD figures were built from public datasets, each under the license on its own dataset card: NVIDIA's Nemotron-SFT-Agentic-v2, Nemotron-SFT-Math-v4, Nemotron-SFT-Science-v2 (CC-BY-SA-4.0), Nemotron-SFT-Competitive-Programming-v2, Nemotron-SFT-Instruction-Following-Chat-v3, Nemotron-SFT-Multilingual-v2, Nemotron-SFT-Long-Context-Transduction-v1 (OpenMDW-1.1), Nemotron-SFT-Safety-v2, Nemotron-Pretraining-Legal-v1 (CC-BY-4.0) and Nemotron-Pretraining-Dataset-sample (Nemotron-CC text; NVIDIA Open Data License Agreement); and Nebius's SWE-agent-trajectories (CC-BY-4.0). Each window's sources are listed in evaluation/eval-windows.json.