Qwen3.6-35B-A3B-MTP-Mixed-Q6_K

Overview

This repository provides a highly optimized, custom-quantized GGUF model of Qwen3.6-35B-A3B MTP, specifically engineered for extreme long-context local inference on Dual-GPU setups (2× RTX 3090, 24GB). The primary objective of this quantization is to maximize Multi-Token Prediction (MTP / Self-Speculative Decoding) efficiency while preserving the reasoning, multimodal, and long-context capacities of the original model.

Unlike my previous Qwen3.6-27B-MTP-Custom-IQ6_K project, this new release is entirely built from the official Unsloth MTP-ready BF16 GGUFs, which already integrate the corrected chat template and proper MTP metadata support. This allowed a much cleaner quantization pipeline while significantly improving speculative decoding behavior and overall throughput.

The architecture uses a carefully mixed quantization strategy:

Core expert up/down tensors quantized to Q6_K
Critical routing, attention, speculative decoding, and output tensors preserved in Q8_0
Token embeddings and output layers preserved in Q8_0
NextN speculative layers fully preserved in Q8_0

This quantization was specifically tuned to maintain extremely high draft acceptance rates while dramatically reducing VRAM usage compared to full Q8_0 deployments.

Base model used for requantization:

unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Engine Compatibility

Important

This model has been specifically designed and validated for the mtp-clean branch of llama.cpp maintained by am17an.

Mainline llama.cpp currently does not achieve comparable MTP performance and still lacks proper multimodal (mmproj) compatibility together with MTP speculative decoding. The mtp-clean branch currently provides:

Proper MTP support
Correct multimodal integration
Much higher speculative decoding throughput
Better tensor splitting behavior on dual GPU systems

Repository used:

git clone -b mtp-clean https://github.com/am17an/llama.cpp.git

As of today, using this branch is strongly recommended to reproduce the performance numbers presented in this repository.

Research & Methodology

Preserving Speculative Decoding (MTP)

Most standard quantization pipelines aggressively compress the speculative decoding layers, severely degrading draft acceptance rates and reducing MTP efficiency.

To avoid this, the following layers were explicitly preserved in Q8_0:

nextn
attention projections
routing tensors
gating tensors
output tensors
embeddings

Meanwhile, the large expert tensors were quantized to Q6_K to reduce memory footprint while preserving quality.

The exact quantization scheme was:

Tensor Pattern	Quantization	Purpose
`token_embd.weight`	`Q8_0`	Preserve embedding fidelity
`output.weight`	`Q8_0`	Preserve final output quality
`blk.[0-9]+.nextn.*`	`Q8_0`	Preserve MTP / speculative decoding layers
`blk.[0-9]+.attn_q.weight`	`Q8_0`	High precision attention query projection
`blk.[0-9]+.attn_k.weight`	`Q8_0`	High precision attention key projection
`blk.[0-9]+.attn_v.weight`	`Q8_0`	High precision attention value projection
`blk.[0-9]+.attn_qkv.weight`	`Q8_0`	Preserve fused attention projections
`blk.[0-9]+.attn_output.weight`	`Q8_0`	Preserve attention output quality
`blk.[0-9]+.ssm_out.weight`	`Q8_0`	Preserve SSM output precision
`blk.[0-9]+.ssm_alpha.weight`	`Q8_0`	Preserve SSM stability
`blk.[0-9]+.ssm_beta.weight`	`Q8_0`	Preserve SSM stability
`blk.[0-9]+.attn_gate.weight`	`Q8_0`	Preserve routing and gating behavior
`blk.[0-9]+.ffn_gate_inp.weight`	`Q8_0`	Preserve FFN routing accuracy
`blk.[0-9]+.ffn_gate_shexp.weight`	`Q8_0`	Preserve shared expert gating
`blk.[0-9]+.ffn_up_exps.weight`	`Q6_K`	Quantized expert expansion tensors
`blk.[0-9]+.ffn_down_exps.weight`	`Q6_K`	Quantized expert reduction tensors
`blk.[0-9]+.ffn_gate_exps.weight`	`Q8_0`	Preserve expert routing precision
`blk.[0-9]+.ffn_up_shexp.weight`	`Q6_K`	Quantized shared expert expansion tensors
`blk.[0-9]+.ffn_down_shexp.weight`	`Q6_K`	Quantized shared expert reduction tensors

This configuration achieved exceptional speculative decoding efficiency even under extremely large context loads.

Performance

Using the recommended server configuration below, the following performance was achieved on a 262K context session:

prompt eval time = 185569.97 ms / 259716 tokens
                 = 0.71 ms per token
                 = 1399.56 tokens per second

eval time       = 29168.45 ms / 2428 tokens
                 = 12.01 ms per token
                 = 83.24 tokens per second

total time      = 214738.42 ms / 262144 tokens

draft acceptance rate = 0.93550
                      = 1581 accepted / 1690 generated

These results represent one of the highest publicly documented MTP acceptance rates currently achieved on a heavily quantized long-context deployment.

Chat Template & Metadata

Unlike my previous Qwen3.6-27B-MTP-Custom-IQ6_K project, this repository no longer required manual GGUF metadata patching.

The model was directly quantized from Unsloth's official MTP-ready BF16 release, meaning:

Correct chat template already integrated
Proper Developer role support
Correct multimodal metadata
Native MTP support
No manual GGUF editing required

iMatrix Calibration

The model was calibrated using a custom shuffled iMatrix designed to preserve performance across:

Coding tasks
Tool usage
Long-context reasoning
English/French bilingual tasks
Instruction following

The calibration dataset was built from merged subsets of:

code_small
tools_small
text_en_small
text_fr_small

from:

eaddario/imatrix-calibration

Recommended Usage

To reproduce the optimal performance characteristics shown above, the following server configuration is recommended:

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Qwen3.6-35B-A3B-MTP-Mixed-Q6_K.gguf \
    --mmproj /path/to/mmproj-F16.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k bf16 \
    --cache-type-v bf16 \
    --context-shift \
    --flash-attn on \
    -b 1024 -ub 1024 \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --spec-draft-p-min 0.75 \
    --ctx-checkpoints 8 \
    --checkpoint-every-n-tokens 65536 \
    --cache-ram 32768 \
    --image-min-tokens 1024 \
    --alias Qwen3.6-35B \
    --jinja

Additional Notes

I also developed a custom proxy layer allowing users to dynamically switch between thinking and non-thinking behaviors while automatically applying the recommended sampling parameters together with "Preserve Thinking" support.

This proxy is available on my GitHub.

Hardware Requirements

Target VRAM: 48 GB (validated on 2× RTX 3090 24GB)
System RAM: Minimum 32GB recommended
Recommended KV cache: F16
Recommended context: 262K

The provided configuration uses a very large KV cache footprint. If you encounter OOM errors, consider:

reducing --ctx-size
lowering batch sizes
using quantized KV cache:
- --cache-type-k q8_0
- --cache-type-v q8_0

Acknowledgments

This project was made possible thanks to the outstanding work of the open-source AI community. Special thanks to:

Unsloth For publishing the official MTP-ready BF16 GGUF release with corrected metadata, proper chat templates, multimodal compatibility, and native speculative decoding support.
am17an For the mtp-clean llama.cpp branch enabling high-performance multimodal MTP inference and substantially improved speculative decoding throughput compared to current mainline llama.cpp.
eaddario For the excellent imatrix-calibration dataset used to build the multilingual and code-oriented custom iMatrix.
The Qwen Team For researching and releasing the exceptional Qwen3.6 architecture and open-weight models enabling this work.

AlexanderKyng/Qwen3.6-35B-A3B-MTP-Mixed-Q8_0