gemma-4-E4B-it-ONNX (text path, q4)
The text half of Gemma 4 E4B instruction-tuned, in ONNX, as consumed by Stenograf's Windows sidecar for on-device question answering over saved meeting transcripts.
Re-hosted from onnx-community/gemma-4-E4B-it-ONNX so the sidecar fetches from one pinned place alongside the other models it downloads. Use of these weights remains subject to the Gemma Terms of Use.
What is here
Only the two graphs the decode loop needs, at q4:
| file | purpose |
|---|---|
onnx/embed_tokens_q4.onnx (+ data) |
ids to hidden state, plus the per-layer inputs this architecture takes |
onnx/decoder_model_merged_q4.onnx (+ data) |
one decode step, 24 key/value pairs in and out |
tokenizer.json |
SentencePiece-style BPE, 262144 pieces |
generation_config.json |
names the stop ids — [1, 106, 50] |
The vision and audio encoders are not included: the transcripts are already text by the time a question is asked.
Why q4 and not q4f16
q4f16 is smaller and looks like the better choice, but it cannot run on
CUDA with ONNX Runtime 1.28. Both variants pass an attention_bias into
GroupQueryAttention, which the CUDA kernel rejects; in q4 those twenty
nodes fall back to the CPU provider and the model still runs, while in fp16 no
such fallback exists and session execution fails outright.
That fallback is also where the remaining headroom is — profiling one decode
step put 23.3% of the time in GroupQueryAttention on the CPU provider. A
runtime that accepts attention_bias on CUDA would recover it for free.
Notes for anyone driving these graphs
- The key/value layers are not uniformly shaped: most carry a 256-wide head, some 512. Build each tensor from its own declared signature.
- Keep the cache on the device between steps. Round-tripping it through host memory measured 2.4x slower on the same GPU.
- Gemma 4 renamed its turn markers to
<|turn>and<turn|>. Gemma 3's<start_of_turn>is absent from this vocabulary, so a prompt written the old way encodes as ordinary text and the model never emits a stop id.