Qwen3.6-27B DSV4Pro Thinking Distill — NVFP4 (multi-tier + official MTP)
NVFP4 quantization of 27B DSV4Pro Thinking Distill with multiple tiers + official nextn MTP head, balancing quality, speed, and concurrency via W4A16 / W4A4 (/ future W4A8) + MTP.
Tiers
| Tier | Path | GPQA-D 198 | MMLU-500 | Size | Focus |
|---|---|---|---|---|---|
| W4A16 | / (root) |
82.83% | 87.80% | 29 GB | quality-first |
| W4A4 | w4a4/ |
77.27% | 91.40% | 20 GB | speed / VRAM |
| W4A8 | pending engine support | ≈ W4A16 (expected) | — | ~19 GB | vLLM loader TBD |
- W4A16 (MLP-focused): only MLP
gate/up/down→ NVFP4; attention, Mamba-GDN, vision, embeddings,lm_head, norms stay BF16. - W4A4: additionally quantizes attention + GDN proj; only conv1d/lm_head/vision/embed/norm/MTP stay BF16.
- Both ship the official MTP head.
MTP (official nextn head) — all speeds @ RTX PRO 6000 Blackwell (sm120), vLLM 0.23
⚠️ Speeds are @R6000 + vLLM. The GGUF repo's MTP numbers are @Spark GB10 + llama.cpp — different hardware, don't compare directly.
| Tier | Recommended num_speculative_tokens |
single-stream tok/s (MTP / no-MTP) | speedup | per-pos accept |
|---|---|---|---|---|
| W4A16 | 4 | 93.7 / 45 | 2.08× | 0.83 |
| W4A4 | 2 | 111 / 63 | 1.76× | 0.87 |
MTP stays net-positive under concurrency (W4A4 N=2, aggregate tok/s): c1 +72% · c4 +58% · c8 +48% · c16 +44%. Gain decreases with concurrency but stays positive through c16 (NVFP4 is memory-bound-wide + sm120 compute-strong; crossover > 16). Use MTP for single-stream / low-to-mid concurrency.
Load
# W4A16 (root) + MTP
vllm serve nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --quantization modelopt \
--max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":4}'
# W4A4 (w4a4/ subdir) + MTP
huggingface-cli download nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --local-dir ./nvfp4
vllm serve ./nvfp4/w4a4 --quantization modelopt \
--max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'
Inference: temperature=0.6, top_p=0.95, max_tokens=32000 (thinking mode). Hardware: Blackwell SM120+ (RTX PRO 6000 / B200). SGLang: --quantization modelopt_fp4 + --speculative-algorithm NEXTN --speculative-num-steps N (optimal num-steps TBD on Spark).
W4A8 (pending vLLM support)
ModelOpt W4A8_NVFP4_FP8_CFG quantizes fine (~19 GB, quality expected ≈ W4A16), and R6000 sm120 has native FP8×FP4 mixed MMA. But vLLM 0.23's ModelOpt loader rejects W4A8_NVFP4_FP8 (accepts only FP8 / FP8_PER_CHANNEL_PER_TOKEN / FP8_PB_WO / NVFP4 / W4A16_NVFP4 / MXFP8 / MIXED_PRECISION). A w4a8/ subdir will be added once an engine supports it.
Eval protocol
temp 0.6 / top_p 0.95 / thinking-on / max_tokens 32000 / ctx 36864; GPQA-Diamond 198 (full) + MMLU-500 5-shot. Quality is hardware-independent; speed numbers are @R6000.
中文说明
NVFP4 多档量化 + 官方 MTP 头，通过 W4A16 / W4A4（/ 未来 W4A8）+ MTP 兼顾质量、速度、并发。
| 档 | 目录 | GPQA-D 198 | MMLU-500 | 大小 | 定位 |
|---|---|---|---|---|---|
| W4A16 | 根 | 82.83% | 87.80% | 29G | 质量优先（仅量 MLP，attention 等保 BF16） |
| W4A4 | w4a4/ |
77.27% | 91.40% | 20G | 速度/显存（量 MLP+attention+GDN，conv1d 保 BF16） |
| W4A8 | 待引擎支持 | ≈W4A16（预期） | — | ~19G | vLLM 暂不支持其 quant_algo |
MTP 峰值 @R6000：W4A16 N=4→93.7 tok/s（2.08×）、W4A4 N=2→111 tok/s（1.76×）。MTP 并发全程正收益（W4A4 c16 仍 +44%，不转负）。所有速度 @ RTX PRO 6000（sm120）vLLM，区别于 GGUF 仓的 Spark GB10 数据。加载命令见上方英文部分。
Claude Code（实验性 / experimental）
Claude Code 支持目前属于实验性接入。Claude Code 需要兼容 Anthropic /v1/messages 的服务端和稳定的工具调用；直接使用 OpenAI-compatible chat endpoint 可能无法正常工作，需要桥接或兼容运行时。
本模型使用 Qwen3 XML 工具调用格式（<tool_call><function=name><parameter=...>）。vLLM 路线：
vllm serve /path/to/model \
--served-model-name qwen36-27b-distill \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml
Claude Code 指向 served-model-name（不要用带 / 的 HF repo id）；或使用 LM Studio 0.4.1+（内置 Claude Code /v1/messages）。参考 vLLM Claude Code · LM Studio。
Claude Code (experimental)
Claude Code support is experimental. It needs an Anthropic-compatible /v1/messages endpoint and stable tool calling; a direct OpenAI-compatible chat endpoint may not work and requires a bridge or compatible runtime. The model uses the Qwen3 XML tool format — on vLLM use --tool-call-parser qwen3_xml (command above). Point Claude Code at the served-model-name (no slashes), or use LM Studio 0.4.1+ (built-in Claude Code /v1/messages).