Gemma4は小さいモデルに先行させる事で推論を高速化するMTP(Multi-Token Prediction)という仕組みがあります。
500MB程度のメモリ使用量が増えますが1.5超の速度向上が期待できます
このモデルはdahara1/gemma-4-12B-it-qat-UD-japanese-imatrixをMTPで動かすための小モデルです。
現時点ではQ4_0またはQ8_0が良さそうに見えます
Gemma4 has a mechanism called MTP (Multi-Token Prediction) that speeds up inference by running a smaller model before the main model.
While memory usage will increase by approximately 500MB, a speed improvement of over 1.5x can be expected.
This model is a small model for running dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix with MTP.
At this point, Q4_0 or Q8_0 looks promising.
Benchmark Reslut (RTX 4060ti(16GB))
私が使用したベンチマークスクリプトは以下です
The benchmark script I used is as follows:
am17an/mtp-bench.py
mtp_bench_jp.py japanese translated version by dahara1.
llama-server command example
./llama-server \
-m ./gemma-4-12B-it-UD-japanese-imatrix/gemma-4-12B-it-qat-ja-UD-Q4_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
--temp 1.0 \
--top-p 0.95 \
--top-k 64 \
--min-p 0.0 \
--ctx-size 32000 \
--flash-attn on \
--jinja \
--model-draft ./gemma-4-423M-12b-it-qat-unquantized-assistant-Q4_0.gguf \
--chat-template-file ./gemma4-12b/chat_template.jinja \
--spec-type draft-mtp \
--reasoning on
Q4_0 mtp_bench.py result.
code_python pred= 192 draft= 185 acc= 129 rate=0.697 tok/s=77.1
code_cpp pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=76.4
explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=77.2
summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=85.3
qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=77.6
translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=72.3
creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=60.1
stepwise_math pred= 192 draft= 165 acc= 136 rate=0.824 tok/s=87.7
long_code_review pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1712,
"total_draft_accepted": 1143,
"aggregate_accept_rate": 0.6676,
"wall_s_total": 24.8
}
Q4_0 mtp_bench_jp.py result.
code_python pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=78.3
code_cpp pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=74.8
explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=71.0
summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=88.3
qa_factual pred= 192 draft= 204 acc= 121 rate=0.593 tok/s=71.1
translation pred= 192 draft= 189 acc= 128 rate=0.677 tok/s=77.2
creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=60.9
stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=73.3
long_code_review pred= 192 draft= 200 acc= 123 rate=0.615 tok/s=70.0
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1782,
"total_draft_accepted": 1116,
"aggregate_accept_rate": 0.6263,
"wall_s_total": 25.83
}
Q8_0 mtp_bench.py result.
code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=75.5
code_cpp pred= 192 draft= 180 acc= 130 rate=0.722 tok/s=77.6
explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=71.0
summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=79.1
qa_factual pred= 192 draft= 181 acc= 130 rate=0.718 tok/s=74.9
translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6
creative_short pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=58.2
stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=81.5
long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=66.8
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1706,
"total_draft_accepted": 1145,
"aggregate_accept_rate": 0.6712,
"wall_s_total": 25.93
}
Q8_0 mtp_bench_jp.py result.
code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=72.8
code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=70.8
explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=66.1
summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=85.4
qa_factual pred= 192 draft= 207 acc= 120 rate=0.580 tok/s=67.3
translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=78.3
creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=59.5
stepwise_math pred= 192 draft= 191 acc= 126 rate=0.660 tok/s=72.5
long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=66.7
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1787,
"total_draft_accepted": 1115,
"aggregate_accept_rate": 0.624,
"wall_s_total": 26.81
}
F16 mtp_bench.py result.
code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.5
code_cpp pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.6
explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=66.3
summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=73.5
qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.9
translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.7
creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.1
stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.6
long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=61.7
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1709,
"total_draft_accepted": 1144,
"aggregate_accept_rate": 0.6694,
"wall_s_total": 28.04
}
F16 mtp_bench_jp.py result.
code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=68.2
code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=65.5
explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=60.3
summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=77.2
qa_factual pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.6
translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=70.7
creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=54.5
stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=65.3
long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=61.2
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1792,
"total_draft_accepted": 1113,
"aggregate_accept_rate": 0.6211,
"wall_s_total": 29.13
}
BF16 mtp_bench.py result.
code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.9
code_cpp pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.8
explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=64.7
summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=72.2
qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.5
translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.3
creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.2
stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.8
long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=60.8
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1709,
"total_draft_accepted": 1144,
"aggregate_accept_rate": 0.6694,
"wall_s_total": 28.22
}
BF16 mtp_bench_jp.py result.
code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=65.9
code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=63.5
explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=59.9
summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=74.1
qa_factual pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.8
translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=71.1
creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=53.0
stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=63.9
long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=60.6
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1792,
"total_draft_accepted": 1113,
"aggregate_accept_rate": 0.6211,
"wall_s_total": 29.63
}