Custom hybrid quant of deepseek-ai/DeepSeek-V4.1-Flash for 4× NVIDIA DGX Spark (GB10).
This is not a uniform 3.5 bpw Pollard EXL3 pack. It is a TR3 hybrid: most routed experts are EXL3 3.0 bpw, the 64 hardest experts per layer stay native MXFP4.
| Piece | Format | BPW |
|---|---|---|
| Routed tail | EXL3-TR3 K=3, codebook mcg | 3.0 |
| Routed keep-64 / layer | native MXFP4 (E2M1 + UE8M0/32) | 4.25 |
| Expert-weighted mix | 64 keep + rest tail | ≈3.22 |
| Attention, shared expert, norms, embeddings | official FP8 / native | unchanged |
| Engram (shards 47–48) | official, hardlinked | unchanged (~203 GB) |
Do not label this repo “3 bpw only.” The honest headline is 3.0 bpw EXL3-TR3 tail + 64 MXFP4 keeps/layer (~3.22 bpw MoE mix).
Disk: official 476 GB → this pack 410 GB (48 shards). Non-Engram 273 → 207 GB (~0.76×), matching 320/384 experts at 3/4.25 and 64/384 kept MXFP4.
Measured on 4x DGX Spark (vLLM, TP4). Reference = native, i.e. the shipped DeepSeek-V4.1-Flash MXFP4-experts/FP8 checkpoint this pack was quantized from (there is no public BF16 original), so KL(native || TR3) is the loss this hybrid adds on top of the release.
KLD is measured to brandonmusic's spec: the teacher's (native's) logits over a representative 32-window x 2047-token corpus = 66,599 teacher-forced positions (>50K), top-20 truncation, KL averaged per position (no alignment artifact).
| metric | native (ref) | TR3-Hybrid | EXL3 3.5bpw Pollard |
|---|---|---|---|
| KL vs native (nats), lower=better | 0 | 0.032 | 0.057 |
| top-1 agreement (teacher-forced) | 1.000 | 0.984 | 0.976 |
| objective battery (30 items) | 24/27 | 25/27 | 25/27 |
Tier-1 (the K3 trellis tail) is high fidelity. TR3's KL is ~44% below the uniform EXL3 3.5 bpw pack, and both track native tightly (top-1 > 97.5%); for scale a third-party GLM-5.3 EXL3 KLD was ~0.102 nats. An early 274-position smoke test read ~0.26 nats and swung with alignment - that sample was far too small (per brandonmusic, use >=50K positions), not a real tier-1 problem.
Full tables, methodology, reproducible scripts and raw captures: https://github.com/drowzeys/keys-DeepSeekV4.1-Flash-TR3-Hybrid-1M-Context-4M-KV-Four-DGX-Sparks
deepseek-ai/DeepSeek-V4.1-Flash (552B backbone, 384 routed + 1 shared, top-6, moe_intermediate_size=2304, 1M context).% 128 == 0).q_fallback (Pass A does not fit 4×128 GB unified memory).encode_dsv41_tr3.py wrapping the TR3 numeric core.config.json field hybrid_tr3_tail: bits: 3.0, codebook: mcg, keep_mxfp4_per_layer: 64.
The hybrid quant method is brandonmusic’s TR3 recipe, first published as brandonmusic/GLM-5.2-NVFP4-TR3-Hybrid: keep the highest round-trip-error experts in native 4-bit, EXL3-TR3 K3 mcg on the tail, same numeric core (encode_tr3_v31.py). This checkpoint ports that method onto DeepSeek-V4.1-Flash. If you use these weights, credit brandonmusic for the hybrid quant.
Also:
This pack is not bot-lab-21/DeepSeek-V4.1-Flash-EXL3-3.5bpw-Pollard (uniform ~3.51 bpw Pollard on all routed experts). Different method, different BPW mix.
EXL3 GEMM requires N % 128 == 0. Uniform TP4 on 2304-wide experts is 2304/4 = 576 (illegal). Options that are legal:
512/640/640/512 (Tony / cuda-exl3 128-aligned split).2304/3 = 768 (legal); needs V4.1 TP3 patches (virtual heads 64→72, etc.).MIT, same as the DeepSeek-V4.1-Flash release.
model-00048-of-00048.safetensors 102 GBmodel-00047-of-00048.safetensors 102 GBmodel-00017-of-00048.safetensors 5.7 GBmodel-00005-of-00048.safetensors 5.7 GBmodel-00011-of-00048.safetensors 5.7 GBmodel-00023-of-00048.safetensors 5.6 GBmodel-00039-of-00048.safetensors 5.6 GBmodel-00031-of-00048.safetensors 5.6 GBmodel-00035-of-00048.safetensors 5.6 GBmodel-00027-of-00048.safetensors 5.6 GBmodel-00018-of-00048.safetensors 5.6 GBmodel-00036-of-00048.safetensors 5.6 GBmodel-00037-of-00048.safetensors 5.6 GBmodel-00042-of-00048.safetensors 5.6 GBmodel-00029-of-00048.safetensors 5.6 GBmodel-00032-of-00048.safetensors 5.6 GBmodel-00014-of-00048.safetensors 5.6 GBmodel-00021-of-00048.safetensors 5.6 GBmodel-00025-of-00048.safetensors 5.6 GBmodel-00026-of-00048.safetensors 5.6 GBmodel-00028-of-00048.safetensors 5.6 GBmodel-00030-of-00048.safetensors 5.6 GBmodel-00015-of-00048.safetensors 5.6 GBmodel-00016-of-00048.safetensors 5.6 GBmodel-00019-of-00048.safetensors 5.6 GBmodel-00022-of-00048.safetensors 5.6 GBmodel-00034-of-00048.safetensors 5.6 GBmodel-00041-of-00048.safetensors 5.6 GBmodel-00013-of-00048.safetensors 5.6 GBmodel-00024-of-00048.safetensors 5.6 GBmodel-00033-of-00048.safetensors 5.6 GBmodel-00038-of-00048.safetensors 5.6 GBmodel-00020-of-00048.safetensors 5.6 GBmodel-00040-of-00048.safetensors 5.6 GBmodel-00004-of-00048.safetensors 5.6 GBmodel-00008-of-00048.safetensors 5.6 GBmodel-00010-of-00048.safetensors 5.6 GBmodel-00003-of-00048.safetensors 5.6 GBmodel-00007-of-00048.safetensors 5.6 GBmodel-00012-of-00048.safetensors 5.6 GBmodel-00006-of-00048.safetensors 5.6 GBmodel-00009-of-00048.safetensors 5.6 GBmodel-00046-of-00048.safetensors 2.7 GBmodel-00044-of-00048.safetensors 2.7 GBmodel-00045-of-00048.safetensors 2.6 GBmodel-00002-of-00048.safetensors 1.3 GBmodel-00043-of-00048.safetensors 1.3 GBmodel-00001-of-00048.safetensors 971 MB有文件?提交种子。
提交种子