4-bit ONNX export of the text decoder of google/gemma-4-12B-it
(revision 43c9d014b231474fa7212dbb48a762566c2b255d), made to run in the browser with
Transformers.js v4 on WebGPU. It powers the “Maximum”
model of WebGPU Chat.
This is a modified version of the original model (Apache 2.0, see LICENSE and NOTICE below):
the weights of the linear layers and the embedding table were quantized to 4 bits, and the vision and
audio towers were removed. It is not an official Google release.
| File | Content |
|---|---|
onnx/embed_tokens_q4f16.onnx (+ .onnx_data) |
token ids → embeddings × √hidden (GatherBlockQuantized, 4-bit) |
onnx/decoder_model_merged_q4f16.onnx (+ .onnx_data … _3) |
48 decoder layers with KV cache → logits |
config.json, generation_config.json, tokenizer*.json, chat_template.jinja |
from the original repository; config.json adapted for Transformers.js (gemma4, text only) |
com.microsoft.MatMulNBits, 4 bits, block 32, asymmetric (zero points), fp16 scales,
round-to-nearest. The tied LM head is split in 4 chunks (≤ 130 MB per GPU buffer).import { AutoTokenizer, Gemma4ForCausalLM } from '@huggingface/transformers';
const id = 'AMINOXDEV/gemma-4-12B-it-ONNX';
const tokenizer = await AutoTokenizer.from_pretrained(id);
const model = await Gemma4ForCausalLM.from_pretrained(id, {
dtype: { embed_tokens: 'q4f16', decoder_model_merged: 'q4f16' },
device: 'webgpu',
});
const inputs = tokenizer.apply_chat_template([{ role: 'user', content: 'Hello!' }], { add_generation_prompt: true, return_dict: true });
const out = await model.generate({ ...inputs, max_new_tokens: 128 });
console.log(tokenizer.decode(out.slice(null, [inputs.input_ids.dims[1], null]), { skip_special_tokens: true }));
Needs a GPU with roughly 10 GB of free memory.
The graph is written node by node by tools/gemma4/export.py
from the official safetensors, read tensor by tensor with HTTP range requests (the math follows
transformers 5.23 modeling_gemma4_unified.py). verify.py
runs this ONNX model with greedy decoding and compares every generated token with the official PyTorch
modules (teacher forcing, equivalent to comparing greedy outputs), both against the original bf16
weights and against the same weights after this 4-bit quantization.
Result (parity.json, 6 prompts in Spanish and English, 274 greedy steps):
| Prompt | Steps | Agreement with original bf16 | Identical greedy prefix | Agreement with same q4 weights | Identical greedy prefix | Mismatch margins (q4 track) |
|---|---|---|---|---|---|---|
| es-chat | 48 | 91.7 % | 10 | 100.0 % | 48 | — |
| en-chat | 34 | 82.3 % | 21 | 94.1 % | 24 | 0.68, 0.279 |
| es-code | 48 | 95.8 % | 0 | 97.9 % | 15 | 0.454 |
| en-code | 48 | 93.8 % | 8 | 95.8 % | 34 | 0.187, 0.101 |
| es-reason | 48 | 89.6 % | 0 | 97.9 % | 1 | 0.057 |
| en-reason | 48 | 97.9 % | 32 | 100.0 % | 48 | — |
“Identical greedy prefix” is how many tokens the greedy outputs share before the first disagreement.
Quantization makes the model somewhat less accurate than the original. Text only (no images or audio). Like any language model it can produce wrong or invented information. See the original model card for intended use and limitations.
Gemma 4 is provided by Google under the Apache License 2.0. Copyright Google LLC. Modifications in this repository (ONNX conversion, 4-bit quantization, removal of the vision and audio towers, Transformers.js configuration) by Amin Tahtah Toumi, also under the Apache License 2.0.
onnx/decoder_model_merged_q4f16.onnx 477.6 kBonnx/embed_tokens_q4f16.onnx 0.9 kBgoogle/gemma-4-12B-it