Gemma4は小さいモデルに先行させる事で推論を高速化するMTP(Multi-Token Prediction)という仕組みがあります。
このモデルはdahara1/gemma-4-E4B-it-UD-japanese-imatrixをMTPで動かすための小モデルです。
Gemma4 has a mechanism called MTP (Multi-Token Prediction) that speeds up inference by running a smaller model before the main model.
This model is a small model for running dahara1/gemma-4-E4B-it-UD-japanese-imatrix using MTP.
llama-server command example
E4Bでは現在、明示的に--flash-attn offにする必要があります。
詳細はissues/24400をご覧ください
In E4B, you now need to explicitly set --flash-attn off.
See issues/24400 for details.
../llama.cpp/build/bin/llama-server \
-m ./gemma-4-E4B-it-UD-Q4_K_XL.gguf \
--host 0.0.0.0 \
--port 8080 \
--temp 1.0 \
--top-p 0.95 \
--top-k 64 \
--min-p 0.0 \
--ctx-size 21000 \
--flash-attn off \
--jinja \
--chat-template-file chat_template.jinja \
--model-draft ./gemma-4-E4B-it-assistant/Gemma-4-E4B-It-Assistant-78M-Q8_0.gguf \
--spec-type draft-mtp \
--reasoning on
ベンチマーク結果 RTX 4060ti(16GB)
ドラフトなしモデル
mtp_bench.py
code_python pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.4
code_cpp pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.7
explain_concept pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.7
summarize pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.5
qa_factual pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.4
translation pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.4
creative_short pred= 35 draft= 0 acc= 0 rate=n/a tok/s=70.4
stepwise_math pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.2
long_code_review pred= 192 draft= 0 acc= 0 rate=n/a tok/s=67.2
Aggregate: {
"n_requests": 9,
"total_predicted": 1571,
"total_draft": 0,
"total_draft_accepted": 0,
"aggregate_accept_rate": null,
"wall_s_total": 24.68
}
mtp_bench_jp.py
code_python pred= 192 draft= 0 acc= 0 rate=n/a tok/s=69.3
code_cpp pred= 192 draft= 0 acc= 0 rate=n/a tok/s=68.0
explain_concept pred= 192 draft= 0 acc= 0 rate=n/a tok/s=65.6
summarize pred= 47 draft= 0 acc= 0 rate=n/a tok/s=65.3
qa_factual pred= 192 draft= 0 acc= 0 rate=n/a tok/s=67.7
translation pred= 192 draft= 0 acc= 0 rate=n/a tok/s=66.9
creative_short pred= 192 draft= 0 acc= 0 rate=n/a tok/s=68.0
stepwise_math pred= 192 draft= 0 acc= 0 rate=n/a tok/s=68.6
long_code_review pred= 192 draft= 0 acc= 0 rate=n/a tok/s=65.7
Aggregate: {
"n_requests": 9,
"total_predicted": 1583,
"total_draft": 0,
"total_draft_accepted": 0,
"aggregate_accept_rate": null,
"wall_s_total": 25.43
}
ドラフトありモデル
code_python pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=157.5
code_cpp pred= 192 draft= 185 acc= 128 rate=0.692 tok/s=153.8
explain_concept pred= 192 draft= 243 acc= 110 rate=0.453 tok/s=121.2
summarize pred= 192 draft= 196 acc= 125 rate=0.638 tok/s=146.6
qa_factual pred= 192 draft= 193 acc= 126 rate=0.653 tok/s=148.7
translation pred= 192 draft= 226 acc= 115 rate=0.509 tok/s=128.1
creative_short pred= 38 draft= 72 acc= 14 rate=0.194 tok/s=81.5
stepwise_math pred= 192 draft= 192 acc= 126 rate=0.656 tok/s=147.5
long_code_review pred= 192 draft= 250 acc= 107 rate=0.428 tok/s=113.9
Aggregate: {
"n_requests": 9,
"total_predicted": 1574,
"total_draft": 1740,
"total_draft_accepted": 981,
"aggregate_accept_rate": 0.5638,
"wall_s_total": 13.48
}
code_python pred= 192 draft= 201 acc= 124 rate=0.617 tok/s=144.8
code_cpp pred= 192 draft= 166 acc= 134 rate=0.807 tok/s=169.0
explain_concept pred= 192 draft= 232 acc= 113 rate=0.487 tok/s=125.7
summarize pred= 51 draft= 54 acc= 33 rate=0.611 tok/s=142.1
qa_factual pred= 192 draft= 191 acc= 127 rate=0.665 tok/s=150.6
translation pred= 192 draft= 225 acc= 115 rate=0.511 tok/s=129.2
creative_short pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=121.1
stepwise_math pred= 192 draft= 204 acc= 123 rate=0.603 tok/s=142.3
long_code_review pred= 192 draft= 221 acc= 117 rate=0.529 tok/s=127.7
Aggregate: {
"n_requests": 9,
"total_predicted": 1587,
"total_draft": 1734,
"total_draft_accepted": 996,
"aggregate_accept_rate": 0.5744,
"wall_s_total": 13.7
}