Qwen3.8-27B-DFlash2-NVFP4-RTNcal

Calibrated NVFP4 (W4A4) quantization of the DFlash 2 block-diffusion draft model for Qwen3.8-27B, packaged in the NVIDIA ModelOpt layout that SGLang loads natively.

Built for a single RTX 5090 (32 GB) serving Qwen3.8-27B in NVFP4 with SGLang: the draft shrinks from 3.53 GB (BF16) to 1.37 GB of VRAM, which turns into +44 % KV-cache context (90K → 130K tokens) at the same decode speed and the same acceptance rate as the BF16 draft. Output quality is unchanged by construction: the target model verifies every drafted token.

* Target gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, FP8 KV cache, --mem-fraction-static 0.90, HiCache on. Benchmark: 5 mixed prompts (code, math, French prose, JSON, essay) × 600 generated tokens, 3 runs each, temperature 0.7. Run-to-run noise on the acceptance length is about ±0.2, so BF16 and this checkpoint are at parity.

Long-context agentic benchmark

Coding-agent session on the SGLang source tree (system prompt + 6 tool schemas, tool results = real source files), thinking on, streaming; 3 turns per context size (cold analysis, follow-up on the cached prefix, tool call with tools), 2 passes. TTFT is identical (prefill is the target's work, ~4K tok/s; cached prefix → 0.04–0.2 s in both), so only decode is shown.

Tool calls were emitted correctly in 10/10 tool turns with this draft (6/8 with BF16, the misses being sampling noise at T=0.7).

What is quantized

• NVFP4 (E2M1, group size 16, FP8-E4M3 block scales, FP32 per-tensor global scale) — all 35 linear projections of the 5 draft layers: self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj. Fused shards (q/k/v, gate/up) share their weight_scale_2 and input_scale, as ModelOpt requires.
• BF16 (unchanged) — fc (target-feature projection), attention_conv.* / mlp_conv.* (dynamic depthwise convolutions), candidate_selector.* (codebooks + hidden projection), all RMSNorm weights. These are plain nn.Linear/parameters in SGLang's DFlash2DraftModel and cannot be quantized there.
• Activations are quantized at runtime by SGLang to NVFP4 with the static per-tensor input_scale stored here (W4A4).

Calibration — the part that matters

Round-to-nearest NVFP4 without calibration loses ~13 % acceptance (3.26 vs 3.71). The whole gap is recovered by calibrating the activation scales on the draft's real inputs:

• On-policy data: 460 conversations generated by the target model itself through SGLang (thinking enabled on half of them), 6 domains — English multi-turn chat (ultrachat), French (Wikipedia-based tasks + free prompts), code (glaive-code-assistant + bash/SQL/TS/Rust/Go/CUDA prompts), tool calling (glaive-function-calling-v2 with tool schemas), math (GSM8K, MATH-500), structured output (JSON/YAML/Markdown). 281,649 generated tokens.
• Capture: forward pre-hooks on the 20 quantizable linears of the BF16 draft while it served those conversations (prefill and decode, eager mode), ~621K input rows per layer. Per-tensor input_scale = amax / (6 · 448).
• Weights: round-to-nearest on the NVFP4 grid; global scale amax / (6 · 448), per-group FP8 scales.

Things that were tried and made no measurable difference on this draft (all within ±0.2 noise): GPTQ with full Hessians, keeping q/k/v in BF16, SmoothQuant folding (o_proj, gate_up, down only — the qkv fold is not function-preserving in SGLang because context K/V are materialized from hidden_norm(fc(features)) without the per-layer input_layernorm).

Usage with SGLang

Requires SGLang built from main at or after commit ecbadf0b (adds DFlash2DraftModel), CUDA 13, a Blackwell GPU (sm120) for the NVFP4 GEMMs, and the draft quantization flag:

sglang serve \
  --trust-remote-code \
  --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --served-model-name qwen3.8-27b \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.90 \
  --attention-backend flashinfer \
  --max-running-requests 1 --cuda-graph-max-bs 1 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 5.61 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-draft-model-quantization modelopt_fp4 \
  --speculative-num-draft-tokens 8 \
  --chunked-prefill-size 1024 \
  --cuda-graph-bs-prefill 64 128 256 512 1024 \
  --enable-memory-saver \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through \
  --host 0.0.0.0 --port 30000

Notes:

• --speculative-draft-model-quantization modelopt_fp4 is required; SGLang does not requantize custom draft models on the fly.
• --speculative-num-draft-tokens 8 is the DFlash 2 block size and cannot be raised.
• The ignore list in config.json (flat ModelOpt format) is what SGLang reads to leave fc & co. in BF16; keep it if you edit the config.
• --cuda-graph-bs-prefill 64 128 256 512 1024 and MAX_JOBS=2 keep host-RAM usage sane on a 64 GB box during graph capture / JIT.
• KV cache in FP4 (--kv-cache-dtype nvfp4) does not work with speculative decoding on this SGLang build (hybrid GDN model).

Files

• model.safetensors — 1,550,153,248 bytes (sha256 2228b9b2…), 186 tensors (35 uint8 packed FP4 weights, 35 FP8 block scales, 70 FP32 scales, 46 BF16).
• hf_quant_config.json, config.json (quantization_config with ignore), tokenizer-free (uses the target's tokenizer), README.incoai-original.md (upstream card).

Reproduce

Scripts (SGLang side): dflash_calib_hook.py (activation capture, temporary patch of sglang/srt/models/dflash.py), calib/build_prompts.py + calib/run_calib.py (on-policy calibration run), quantize-dflash2-gptq-nvfp4.py --rtn (export). Base weights: incoai/Qwen3.8-27B-DFlash2 (BF16). Quantized on 2026-08-29.

Changelog

• 2026-08-30 — config.json: quantization_config.producer is now ModelOpt's dict form (was a plain string, which crashed vLLM's draft ModelConfig construction with AttributeError: 'str' object has no attribute 'get'; SGLang ignored the key). Weights unchanged (same model.safetensors, sha256 2228b9b2…). Thanks to @joelafrite for the report and the vLLM numbers.
• 2026-08-29 — initial release.

License

Apache-2.0, same as the base DFlash 2 draft and Qwen3.8-27B.