Extracted MTP head from Qwen3.5-9B
This is an extracted MTP and quantized Multi Token Prediction (MTP) head from the official Qwen/Qwen3.5-9B. It can be used with models that have been derived from Qwen3.5 9B but do not contain an MTP head themselves anymore such as Dharmamitra's Qwen 3.5 Translate and Dharmamitra's Qwen 3.5 IT models
Multi Token Prediction can significantly improve generation speed when producing languages that the MTP part of the model is familiar with, such as English. The quality of the result remains identical compared to the standalone model without MTP because each predicted token is validated against the main model before being accepted.
Note: this is ONLY the MTP head to perform token prediction (speculative decoding). It needs to be combined with a Qwen3.5 9B based base model in order to function. The official release of Qwen3.5 9B already contains this MTP head so no extra download is needed there. But if you have a Qwen3.5 9B based model that comes without the MTP head then you can use this model file as an addition to restore the availability of MTP.
llama-cli example
Add the following additional parameters to your llama-cpp (llama-cli or llama-server) commandline to load
the MTP model in addition to your regular model and have it speculatively predict tokens:
--spec-type draft-mtp \
--model-draft /path/to/mtp-Qwen3.5-9B-Q8_0.gguf \
--spec-draft-n-max 3 \
--spec-draft-p-min 0.75
Credit, acknowledgement and license
- Base model: Qwen/Qwen3.5-9B.
- License: Apache-2.0, inherited from the original release. These GGUF files are requantized versions of the same weights; no change of license is intended or implied.