canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP

A DeepSeek-V4-Flash NVFP4-FP8 quantization that retains the MTP (multi-token-prediction) block in the saved weights, so vLLM can load it with --speculative-config method=mtp.

🛑 Hardware scope — B200 / B300 only

This artifact targets datacenter Blackwell (SM 10.0 / sm_100a / sm_103a) — the only hardware where vLLM's native NVFP4 tensor-core path (tcgen05) actually executes the MoE matmul in FP4.

On consumer Blackwell (RTX PRO 6000 SE / DGX Spark / RTX 5090, SM 12.0+) vLLM falls back to the Marlin BF16 kernel which dequantizes FP4 → BF16 inside the kernel. The fallback works correctly but forfeits the FP4 FLOPS advantage that motivates NVFP4 in the first place, and on the same RTX PRO 6000 hardware the W4A16 sibling has a smaller footprint (~159 GB vs 172 GB) and slightly higher throughput on its native Marlin INT4 path.

If you're deploying on RTX PRO 6000, DGX Spark, RTX 5090, or any consumer Blackwell, use the W4A16 sibling instead:

• canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP

What this is

• 172 GB across 35 safetensors shards (vs ~600 GB BF16 source, MTP block included).
• Same quantization scheme as RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8: NVFP4 (group=16, FP8 e4m3 scales) on routed FFN experts, FP8_BLOCK 128×128 on attention.
• MTP block (mtp.0.*, 799 tensors) kept at BF16 — not dropped at load time, not double-quantized when the MTP draft model is constructed.

That last point is the only structural difference from RedHat's artifact. The HF transformers DSV4 modeling class has _keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"], which silently strips MTP keys during the calibration load path. RedHat's artifact ran through that path, so their saved weights don't include MTP. Ours include MTP because we patched the modeling class during calibration.

Hardware validated

Server with CUDA graphs enabled. RTX PRO 6000 / DGX Spark / RTX 5090 are out of scope for this artifact — use the W4A16 sibling for those.

Accuracy (4× B300, TP=4)

Measured 2026-05-21. Same prompts, temperature 0, chat template server-side.

On raw AIME pass@1, RedHat scores higher than ours (90% vs 83%) — but the gap is entirely truncation rate at the 65K max_tokens cap (RedHat truncated 2/30, ours truncated 5/30). On non-truncated pass@1 — the problems where both completed reasoning — all three configs are within 0.4pt of each other (96.0–96.4%). Quantization quality is equivalent across configs on AIME 2024 in this measurement; the differentiator is wall-clock.

Throughput (4× B300 SXM6, SM 10.3, NVLink, TP=4)

Same hardware, same TP=4, same prompts as the accuracy table above.

Two different ratios to disambiguate:

• Pure decode throughput: at c=1 chat coding, ours is 2.13× faster. On AIME reasoning at c=8, the per-request median decode rate is 182.9 vs 99.6 tok/s — a 1.84× decode speedup.
• AIME batch wall-clock: 1405s / 476s = 2.95×. This includes the truncation-rate differential at the 65K max_tokens cap.

MTP draft acceptance per workload (B300)

Acceptance does not degrade under batching — flat at 88.0% ± 0.4% across c=1 to c=16 on chat-templated coding.

Recommended serve config (B300, TP=4)

VLLM_TEST_FORCE_FP8_MARLIN=1 \
vllm serve canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP \
    --tensor-parallel-size 4 \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --tokenizer-mode deepseek_v4 \
    --reasoning-parser deepseek_v4 \
    --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
    --trust-remote-code \
    --host 0.0.0.0 --port 8000

--tensor-parallel-size 4 on B300; TP=4 is faster than TP=8 on this artifact because the per-rank MoE expert shard is small enough that the additional TP-allreduce overhead beats the extra compute. TP=4 was 6-22% faster than TP=8 at c≥4 in our measurements.

NVFP4 hardware reality (the bit users keep asking about)

Until CUTLASS#3096 (sm_120a NVFP4 grouped-GEMM garbage) is fixed AND vLLM's NVFP4 oracle is extended to allow sm_120 backends AND flashinfer_cutedsl_sm12x learns to apply SwiGLU clamp — three independent upstream tracks — NVFP4 on consumer Blackwell is storage-only, not compute. Use W4A16 for those platforms.

License

MIT, inherited from upstream deepseek-ai/DeepSeek-V4-Flash.

Reproduction

GitHub: https://github.com/canada-quant/dsv4-flash-nvfp4-fp8-mtp — calibration scripts, B300 install path, and the open-PR queue tracking upstream contributions extracted from this work.