DeepSeek-V4.1-Flash-1017 · dsx T16

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) in one 252 GB download for two RTX 3090s and 128 GB of RAM, with the routed experts in T16, dsx's own trellis format: exllamav3's bitshift trellis with a codebook made for desktop CPUs, quantized from DeepSeek's original weights. Runs on dsx, a Rust + CUDA engine for this model.

The default package contains an IQ4_NL Engram table. The optional q6k-engram branch uses Q6_K instead; the quality measurements below identify the package used for each result.

Release measurements

The quality and speed figures below belong to this model release and its recorded runtime. They are not a new measurement of every later kernel revision. See NVIDIA measurement status for the runtime settings and open correctness checks, and model release history for other packages.

At a glance

Quality

Measured on Lucebox's 1017 eval set: 8 x 1024 tokens against the FP8 model's top-256 log-probs. T16 and dsx-ud were measured in the same session and build.

Historical release measurements through the decode path (one token at a time; 2 x 1,023 tokens of the same set, without deferral). Later integrated-kernel checks have a separate unresolved KL shift; see the status link above:

How T16 works

The experts are quantized with exllamav3's machinery (a QTIP-style bitshift trellis found by a Viterbi search, 128-wide Hadamard rotations, LDLQ error feedback with each expert's input Hessian), with five changes:

1. The T16 codebook. A weight's value comes from its 16-bit window of the bit stream: multiply by 0x3A2F (16 bits) and add the product's two bytes. That is one vpmullw and one vpdpbusd per pair of weights on an AVX-VNNI CPU, where exllamav3's codebook needs a 32-bit multiply per weight. ~4% more error than exllamav3's on random data.
2. Column-major positions inside each 16 x 16 tile, so one file decodes fast on both the CPU and the GPU.
3. A damped Hessian (sigma_reg 2). With the full Hessian the experts fit the calibration text but lose 1.6-2.4x of their accuracy on new text; damping keeps the fit general.
4. An output-gain correction. A trellis quantizer that minimizes squared error reconstructs weights about 13% too small at 1.5 bits (7% at 2 bits): shrinking cuts the noise more than it adds bias. Per matrix that is optimal, but the bias is the same in every expert and layer, so it compounds through the network. Each output's stored scale is multiplied back (x1.13 for 1.5-bit gate/up, x1.04 for 2-bit gate/up); on a half-T16 model that took KL from 0.31 to 0.25. It costs nothing at run time. The two values were chosen on the eval set above.
5. The bit layout above, sized to match Unsloth's UD-IQ1_M.

Calibration: 635K tokens (coding-agent sessions, agent and code prompts, the model's own reasoning traces, Wikipedia in five languages, code, chat). The calib5 requant completed on 575×2048 = 1,177,600 fit tokens. It scored KL ~0.202 and top-1 81.3% with v1's gains reused, versus v1's 0.196 / 80.9%, so it did not replace v1. Agentic-quality gains were not established. The gains were selected on this eval; independent gain tuning remains unmeasured.

October 25 runtime checkpoint

The same T16 weights reached 1,007.84 newly processed prompt tokens/s on two RTX 3090s, an i7-12700K and 128 GB DDR4. The benchmark uses an 8,000-token prompt, an 8,192-token chunk, a 256K context allocation and fixed expert placement. It resets prompt state between rounds: the two warm passes measured 1,008.50 and 1,007.18 tok/s, with no reused prompt KV. The initial pass measured 928.49 tok/s. F16 Engram projections remain unchanged.

The opt-in profile combines warp Hadamard transforms, fixed-bit weight expansion, fused gate/up post-processing and prefill buffer reuse. Buffer reuse recovers about 1.91 GB of GPU memory at this chunk size. The matched 8K-prompt baseline with 4K chunks measured 868.07 tok/s; the final experimental screen measured 1,006.43 tok/s, about 16% faster. Its 64-token forced-decode continuations measured about 40.4 tok/s. That short decode check is separate from the 41 tok/s free-generation result with optional FP8 projections.

See the profile, code and receipts for the flags and complete conditions. The new options are off by default. Kernel audits cover 84 exact cases; the main integration also matches 129,280 final logits bit for bit. These are implementation checks, not a new FP8-reference quality score or a resolution of the earlier open decode check. Smaller prompts and cached tool updates have different throughput.

Speed

2x RTX 3090 (250 / 280 W) + i7-12700K + 128 GB DDR4, dsx-ud measured in the same session:

The free-running rows generate different text per build, which moves the share of experts the CPU computes; the teacher-forced row (identical tokens) is the fair comparison: 15% faster. Prefill was ~9% slower at first; dsx since expands T16 weights 2.5-3x faster (899e6e4) and keeps the RAM arena in hugepages, which copy to the GPU at 24.6 GB/s instead of 13.5 (cc88c1a): 3K-token prefill 388 → 529 tok/s on the same build otherwise. dsx-ud was not re-measured.

Per layer, T16's experts compute 20-34% faster than Unsloth's codebook mix on the i7-12700K (14 threads: 1.21 / 1.46 ms per 6-expert call vs 1.83) and 22% faster on an RTX 3090 (125 vs 161 us per layer's 6 experts).

Run

hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-t16 --local-dir dsx-t16

git clone https://github.com/lxe/dsx && cd dsx
CUDA_ARCHS=86 cargo build --release -p dsx --features cuda      # 86 = RTX 30xx, 120 = RTX 50xx

./target/release/dsx-server --model ../dsx-t16/DeepSeek-V4.1-Flash-1017-dsx-t16.gguf \
  --host 0.0.0.0 --port 8090 --ctx 131072 --arena-gb 110 --rank ../dsx-t16/expert-rank.json --defer-rank 3

The Q6_K Engram version (302 GB, KL 0.194 / top-1 81.4% instead of 0.196 / 80.9%) is on the branch q6k-engram:

hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-t16 --revision q6k-engram --local-dir dsx-t16-q6k
# then --model dsx-t16-q6k/DeepSeek-V4.1-Flash-1017-dsx-t16-00001-of-00004.gguf (4 shards; dsx loads the rest)

The Engram table and the tokenizer are read from the GGUF. Leave out --defer-rank 3 to avoid delayed expert injection. In a later fixed-token T16 decode eval, R=3 raised KL from ~0.208 to 0.237; those decode/deferral figures use a different protocol from the prefill quality table. Open http://localhost:8090/ for the web UI, or point any OpenAI-compatible client at http://localhost:8090/v1.

Hardware

Files

Credits

DeepSeek for the model (MIT) · exllamav3 (turboderp, MIT) for the trellis quantizer that T16 patches · the T16 format: docs/t16.md · QTIP (Tseng et al.) for bitshift trellises · llama.cpp / ggml for the Q6_K and IQ4_NL formats (MIT) · Unsloth for the IQ4_NL Engram table · Xolotl / Strata for the V4.1 GGUF format (MIT) · Lucebox for the eval set.