DeepSeek V4 Flash — REAP K150 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K150 (150 routed experts per MoE layer, 41% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121).

At a Glance

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2_XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s). Attention and shared experts stay Q8_0 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

Inference (DGX Spark / NVIDIA GB10)

# Managed memory required — saves ~10 GiB RAM, costs 1-3 t/s
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf --ctx 131072

# With FP8-packed KV cache
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf --ctx 262144

API Server

DS4_CUDA_MANAGED_MODEL=1 ./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 131072

Memory Budget (DGX Spark, 128 GB unified memory)

K150 needs managed memory (DS4_CUDA_MANAGED_MODEL=1) to avoid duplication. This saves ~10 GiB of RAM at the cost of 1-3 t/s lower throughput. 1M context requires FP8-packed KV (DS4_KV_TURBO=1).

How It Was Built

1. Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-150 per layer (layers 3–42), layers 0–2 preserved at 256
3. NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
4. Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Variants

Acknowledgments

• DeepSeek — DeepSeek V4 Flash base model
• antirez — ds4 engine and GGUF quants
• Cerebras Research — REAP expert pruning
• NVIDIA — DGX Spark (GB10) hardware
• eouya2 — reap-for-ds4 tooling