Qwen3.6-27B DSV4Pro Thinking Distill — NVFP4 (multi-tier + official MTP)

NVFP4 quantization of 27B DSV4Pro Thinking Distill with multiple tiers + official nextn MTP head, balancing quality, speed, and concurrency via W4A16 / W4A4 (/ future W4A8) + MTP.

Tiers

Tier	Path	GPQA-D 198	MMLU-500	Size	Focus
W4A16	`/` (root)	82.83%	87.80%	29 GB	quality-first
W4A4	`w4a4/`	77.27%	91.40%	20 GB	speed / VRAM
W4A8	pending engine support	≈ W4A16 (expected)	—	~19 GB	vLLM loader TBD

W4A16 (MLP-focused): only MLP gate/up/down → NVFP4; attention, Mamba-GDN, vision, embeddings, lm_head, norms stay BF16.
W4A4: additionally quantizes attention + GDN proj; only conv1d/lm_head/vision/embed/norm/MTP stay BF16.
Both ship the official MTP head.

MTP (official nextn head) — all speeds @ RTX PRO 6000 Blackwell (sm120), vLLM 0.23

⚠️ Speeds are @R6000 + vLLM. The GGUF repo's MTP numbers are @Spark GB10 + llama.cpp — different hardware, don't compare directly.

Tier	Recommended `num_speculative_tokens`	single-stream tok/s (MTP / no-MTP)	speedup	per-pos accept
W4A16	4	93.7 / 45	2.08×	0.83
W4A4	2	111 / 63	1.76×	0.87

MTP stays net-positive under concurrency (W4A4 N=2, aggregate tok/s): c1 +72% · c4 +58% · c8 +48% · c16 +44%. Gain decreases with concurrency but stays positive through c16 (NVFP4 is memory-bound-wide + sm120 compute-strong; crossover > 16). Use MTP for single-stream / low-to-mid concurrency.

Load

# W4A16 (root) + MTP
vllm serve nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --quantization modelopt \
  --max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":4}'

# W4A4 (w4a4/ subdir) + MTP
huggingface-cli download nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --local-dir ./nvfp4
vllm serve ./nvfp4/w4a4 --quantization modelopt \
  --max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

Inference: temperature=0.6, top_p=0.95, max_tokens=32000 (thinking mode). Hardware: Blackwell SM120+ (RTX PRO 6000 / B200). SGLang: --quantization modelopt_fp4 + --speculative-algorithm NEXTN --speculative-num-steps N (optimal num-steps TBD on Spark).

W4A8 (pending vLLM support)

ModelOpt W4A8_NVFP4_FP8_CFG quantizes fine (~19 GB, quality expected ≈ W4A16), and R6000 sm120 has native FP8×FP4 mixed MMA. But vLLM 0.23's ModelOpt loader rejects W4A8_NVFP4_FP8 (accepts only FP8 / FP8_PER_CHANNEL_PER_TOKEN / FP8_PB_WO / NVFP4 / W4A16_NVFP4 / MXFP8 / MIXED_PRECISION). A w4a8/ subdir will be added once an engine supports it.

Eval protocol

temp 0.6 / top_p 0.95 / thinking-on / max_tokens 32000 / ctx 36864; GPQA-Diamond 198 (full) + MMLU-500 5-shot. Quality is hardware-independent; speed numbers are @R6000.

中文说明

NVFP4 多档量化 + 官方 MTP 头，通过 W4A16 / W4A4（/ 未来 W4A8）+ MTP 兼顾质量、速度、并发。

档	目录	GPQA-D 198	MMLU-500	大小	定位
W4A16	根	82.83%	87.80%	29G	质量优先（仅量 MLP，attention 等保 BF16）
W4A4	`w4a4/`	77.27%	91.40%	20G	速度/显存（量 MLP+attention+GDN，conv1d 保 BF16）
W4A8	待引擎支持	≈W4A16（预期）	—	~19G	vLLM 暂不支持其 quant_algo

MTP 峰值 @R6000：W4A16 N=4→93.7 tok/s（2.08×）、W4A4 N=2→111 tok/s（1.76×）。MTP 并发全程正收益（W4A4 c16 仍 +44%，不转负）。所有速度 @ RTX PRO 6000（sm120）vLLM，区别于 GGUF 仓的 Spark GB10 数据。加载命令见上方英文部分。

Claude Code（实验性 / experimental）

Claude Code 支持目前属于实验性接入。Claude Code 需要兼容 Anthropic /v1/messages 的服务端和稳定的工具调用；直接使用 OpenAI-compatible chat endpoint 可能无法正常工作，需要桥接或兼容运行时。

本模型使用 Qwen3 XML 工具调用格式（<tool_call><function=name><parameter=...>）。vLLM 路线：

vllm serve /path/to/model \
  --served-model-name qwen36-27b-distill \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Claude Code 指向 served-model-name（不要用带 / 的 HF repo id）；或使用 LM Studio 0.4.1+（内置 Claude Code /v1/messages）。参考 vLLM Claude Code · LM Studio。

Claude Code (experimental)

Claude Code support is experimental. It needs an Anthropic-compatible /v1/messages endpoint and stable tool calling; a direct OpenAI-compatible chat endpoint may not work and requires a bridge or compatible runtime. The model uses the Qwen3 XML tool format — on vLLM use --tool-call-parser qwen3_xml (command above). Point Claude Code at the served-model-name (no slashes), or use LM Studio 0.4.1+ (built-in Claude Code /v1/messages).