DeepSeek V4 Flash — REAP K128 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K128 (128 routed experts per MoE layer, 50% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121).

At a Glance

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2_XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s). Attention and shared experts stay Q8_0 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

Inference (DGX Spark / NVIDIA GB10)

# Basic usage — K128 fits comfortably without managed memory
./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With managed memory — saves ~10 GiB RAM, costs 1-3 t/s (optional for K128)
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With FP8-packed KV cache (save ~7 GiB at 1M ctx)
DS4_KV_TURBO=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 1048576

API Server

./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 131072

Memory Budget (DGX Spark, 128 GB unified memory)

K128 is the most comfortable fit on 128 GB — ample headroom for KV cache growth.

How It Was Built

1. Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-128 per layer (layers 3–42), layers 0–2 preserved at 256
3. NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
4. Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Why REAP + Hybrid NVFP4?

Standard GGUF quants (IQ2_XXS, Q2_K, Q4_K) are compact but sacrifice precision at low bit widths. NVFP4 trades some speed for substantially better quality: it reads more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4 mapping, meaning no requantization error on expert weights. The result is a model that prioritizes precision over raw throughput — slightly slower decode than an all-IQ2_XXS quant, but with dramatically better weight fidelity on the expert path.

Combined with REAP (50% expert pruning), the result is a model that:

• Fits comfortably on a single DGX Spark (128 GB)
• Runs 1M context with FP8-KV
• Preserves attention quality (Q8_0)
• Has no quantization quality loss on experts (lossless MXFP4→NVFP4)

Variants

Acknowledgments

• DeepSeek — DeepSeek V4 Flash base model
• antirez — ds4 engine and GGUF quants
• Cerebras Research — REAP expert pruning
• NVIDIA — DGX Spark (GB10) hardware
• eouya2 — reap-for-ds4 tooling