Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — GGUF (native MTP + imatrix)

Lynn Agent high-end local option

This GGUF repo is the high-end local model option for Lynn Agent. Lynn v0.85.6+ recommends the 27B Q5_K_M GGUF as the default local path and exposes this 35B-A3B Q5_K_M GGUF for 32GB+ VRAM / unified-memory machines.

• Download Lynn Agent: GitHub Releases v0.85.6
• High-end quant in Lynn: Q5_K_M imatrix + native MTP (--spec-draft-n-max 3)
• Default local path: 27B Q5_K_M GGUF for 24GB+ machines; 9B / 4B remain manual low-config downgrades.

GGUF quantizations of nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-35B-A3B (MoE, 35B total / ~3B active). Every GGUF carries a native MTP head and is imatrix-corrected (except Q8_0/F16). The MoE architecture plus MTP speculative decoding gives an especially strong single-stream speedup (below).

Quantizations

Evaluation — distill vs base, Q4_K_M GGUF, same harness, thinking-on

GPQA jumps +7.6pp and orchestration is 2.3× faster — the distill stops over-thinking and converges (non-terminating empties 12→1). MMLU dips 1.2pp: the 35B-A3B has less spare capacity than the 27B dense, so distillation slightly crowds out knowledge (a known trade-off).

MTP — native multi-token-prediction (MoE → strong speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp). MoE benefits more from speculative decoding than dense. Single-stream measured on DGX Spark GB10:

Higher quant tiers gain more (the bandwidth-bound base is accelerated by a cheap draft). Lossless (greedy speculative). Concurrency degrades speculation — use bare multi-concurrency for throughput.

Run

llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

Thinking model — use temp 0.6 / top_p 0.95. For image input add --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf.

---

中文

nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-35B-A3B(MoE,35B 总参 / 约 3B 激活)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了原生 MTP 头,并做了 imatrix 校正(Q8_0/F16 除外)。MoE 架构 + MTP 投机解码,单流加速尤其明显(见下)。

量化档位

评测 —— 蒸馏 vs 原版,Q4_K_M GGUF,同 harness,thinking-on

GPQA +7.6pp、编排快 2.3×(学会收口,空答 12→1)。MMLU 微降 1.2:35B-A3B 容量比 27B dense 紧,蒸馏轻微挤占知识(已知权衡)。

MTP —— 原生多 token 预测(MoE → 加速更猛)

GGUF 内含原生 MTP 头(mainline llama.cpp --spec-type draft-mtp)。MoE 比 dense 更吃投机解码红利。DGX Spark GB10 单流实测:

越高档加速越明显(带宽受限的裸推理被廉价草稿加速)。无损(贪心投机)。并发会摊薄投机 —— 高吞吐用裸多并发。

运行

llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

思考型模型 —— 必须 temp 0.6 / top_p 0.95。图像输入加 --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf。

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。GGUF 走 LM Studio：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 FP8 仓 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3，模型名用 --served-model-name（不带斜杠）。参考 LM Studio · vLLM Claude Code。

---

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. GGUF goes via LM Studio: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the FP8 repo or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3, model name = --served-model-name (no slashes).