Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8

🇬🇧 English · 🇨🇳 中文 ⬇️

---

🇬🇧 English

The FP8 (block-128 e4m3) + native MTP build of Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — for SGLang / vLLM serving. 35B-A3B is MoE (256 experts, 3B active); sparse activation makes it fast even single-stream.

For distillation method, attribution, eval methodology, and limitations, see the parent (BF16) card. This card covers only the FP8 build.

What this is

• FP8 block-128, e4m3, dynamic activation quant of the distilled backbone — same scheme as official Qwen/Qwen3.6-35B-A3B-FP8 (one weight_scale_inv per 128×128 block). All 256 experts quantized per-expert (gate/up/down split to match the official layout, loads directly in SGLang/vLLM). norm, embedding, lm_head, router, Gated-DeltaNet small projections, and the vision tower stay BF16.
• Bundles the native MTP (nextn) head (mtp.safetensors) for EAGLE/NEXTN speculative decoding.
• Near-lossless by construction — block-128 FP8 is Qwen's own "virtually identical to the original" precision.

Speed — this distilled FP8, measured on DGX Spark (GB10, Blackwell sm_121), SGLang dev-cu13 + flashinfer + EAGLE/NEXTN

~1.25× (+25%) single-stream speedup from MTP (base ~51 → 63–65 tok/s, stable accept ~0.75 / accept len ~3.0). All measured on this distilled FP8 (SGLang dev-cu13, EAGLE topk=1 / 3 draft / mamba extra_buffer): correct generation, no FP8 corruption.

Modest MTP gain, but already-fast single-stream: the MTP single-stream gain is modest (~25%) — A3B is sparse and the base is already fast, so speculation headroom is limited (density rule: the sparser the activation, the lower the speculative-decoding ROI). But the native nextn head is cheap enough to stay net-positive (unlike an external drafter, which loses on sparse models). The real payoff is that sparsity makes single-stream fast: 35B-A3B (3B active) 51→65 tok/s, far above the dense 27B (~15 tok/s). Faster on Blackwell RTX-50 (faster FP8).

Quality

FP8 is near-lossless and smoke-verified correct on GB10 (EN/ZH reasoning, intact <think> + closure, no FP8 corruption). Quality ≈ parent distill, whose canonical (thinking-on) numbers:

Multimodal note

The multimodal architecture is inherited from the original base (the vision tower is the stock official one, bundled with the FP8 as outside.safetensors). Distillation targeted text reasoning / agentic only, not vision. The FP8 build supports SGLang/vLLM multimodal; GGUF is text-only (vision via the separate mmproj in the parent repo's gguf/).

Usage (SGLang + MTP)

docker run --gpus all --network host lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
  --model-path nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer --attention-backend flashinfer \
  --trust-remote-code --mem-fraction-static 0.75

Sampling: temperature 0.6 / top_p 0.95 (required for thinking models; greedy loops).

---

🇨🇳 中文版

Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 的 FP8(block-128 e4m3)+ 原生 MTP 版 —— 面向 SGLang / vLLM 服务化。35B-A3B 是 MoE(256 专家,3B 激活),稀疏激活让它单流就很快。

蒸馏方法、出处归因、评测口径、局限见父模型(BF16)卡片。本卡只讲 FP8 版。

这是什么

• FP8 block-128、e4m3、动态激活量化蒸馏主干 —— 与官方 Qwen/Qwen3.6-35B-A3B-FP8 同方案(每 128×128 块一个 weight_scale_inv)。256 个专家逐专家(per-expert)量化(gate/up/down 拆分对齐官方布局,SGLang/vLLM 直认)。norm、embedding、lm_head、router、Gated-DeltaNet 小投影、shared_expert 之外的 BF16 项、vision 塔保持 BF16。
• 打包原生 MTP(nextn)头(mtp.safetensors),供 EAGLE/NEXTN 投机解码。
• 构造上近无损 —— block-128 FP8 正是 Qwen 官方"性能与原模型几乎一致"的同精度。

速度 —— 蒸馏 FP8 实测 DGX Spark(GB10,Blackwell sm_121),SGLang dev-cu13 + flashinfer + EAGLE/NEXTN

MTP 单流约 1.25×(+25%)加速(base ~51 → 63–65 tok/s,接受率稳定 ~0.75 / 接受长度 ~3.0)。均为本蒸馏 FP8 实测(SGLang dev-cu13,EAGLE topk=1 / 3 draft / mamba extra_buffer):生成正确、无 FP8 损坏。

增益温和但单流本就快:MTP 增益温和(~25%) —— A3B 稀疏、base 本已快,投机空间小(密度律:激活越稀疏,投机解码 ROI 越低)。但原生 nextn 够便宜,仍净正收益(不像外挂 drafter 在稀疏模型上反亏)。真正红利是稀疏让单流就快:35B-A3B(3B 激活)51→65 tok/s 远超 dense 27B(~15 tok/s)。Blackwell RTX-50(FP8 更快)上更快。

质量

FP8 近无损,且在 GB10 上冒烟验证正确(中英推理、<think> + 收口结构完整,无 FP8 损坏)。质量 ≈ 父蒸馏版,其 canonical(thinking-on)数据:

知识广度基本不变,硬推理显著提升(GPQA +7.6pp),且"想完就收口"(空答 12→1)。

多模态说明

35B-A3B 的多模态架构继承自原版 base(vision 塔为官方原版,已随 FP8 打包,outside.safetensors)。蒸馏仅针对文本推理 / agentic,未训练视觉。FP8 版可走 SGLang/vLLM 多模态;GGUF 纯文本(视觉单独走 父仓 gguf/ 里的 mmproj 文件)。

用法(SGLang + MTP)

docker run --gpus all --network host lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
  --model-path Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer --attention-backend flashinfer \
  --trust-remote-code --mem-fraction-static 0.75

采样:temperature 0.6 / top_p 0.95(thinking 模型必需,greedy 会重复死循环)。

Claude Code（实验性 / experimental）

Claude Code 不是本地 GGUF/HF 加载器——它对接兼容 Anthropic /v1/messages 的后端、且要求可靠的工具调用，不能直接喂仓名/路径。本 FP8 仓是 vLLM 路线的直接载体：

vllm serve nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --served-model-name qwen36-35b-fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Claude Code 里模型名填 qwen36-35b-fp8（即 --served-model-name，不要填带 / 的仓名），ANTHROPIC_BASE_URL 指向你的 vLLM 端点。要 GGUF + LM Studio 路线见 GGUF 仓。参考 vLLM Claude Code。

---

Claude Code (experimental)

Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible /v1/messages backend and needs reliable tool calling, so you cannot point it at a repo id directly. This FP8 repo is the direct vLLM path — serve it with --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 (command above), then set Claude Code's model name to the --served-model-name (no slashes) and ANTHROPIC_BASE_URL to your vLLM endpoint. For the GGUF + LM Studio route, see the GGUF repo.