Usage

For now, you need to use the gg/spec-mtp-experiments branch on llama.cpp or a custom mtp fork.

You can switch to the mtp branch with git checkout gg/spec-mtp-experiments after cloning and entering the llama.cpp repository.

Add --spec-type mtp --spec-draft-n-max 5 --spec-draft-n-min 0 to your llama-server or llama-cli command.

Feel free to tweak --spec-draft-n-max and find out what works best for your setup.

Try not to push --spec-draft-n-min too far, keep it in single digits.

Credits

• Qwen for this amazing model.
• Unsloth for the imatrix quantization file.
• All of the llama.cpp and ggml contributors for allowing me to run AI models locally.