gemma-4-E2B-it, Q4_0/Q4_K mixed (GGUF)

Release: built to be downloaded and served with Xyntetik Runner.

• Measured status against the parent: fails the fidelity bar (the dual-column numbers are on this card); published as the quickstart file for an 8 GB machine, not as a fidelity claim.
• Parent model: google/gemma-4-E2B-it
• Collection: Runner Releases
• Evidence: the tables on this card
• Runner compatibility: the file the Runner README quickstart names.

Serve with xyntetik-runner — tool calls that still parse when the budget runs out

This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:

When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.

Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models — tight context, slow generation — that is the difference between an agent loop that finishes and one that retries from scratch.

Measured — what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1→64):

Every engine completes at 64 tokens — the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. † TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.

Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) — it guarantees the shape of a call at any quantization, not its contents.

Quickstart

# build the single binary (no deps), then serve this GGUF:
runner -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve
# point any OpenAI-compatible client at http://localhost:8080/v1

Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.

Run it in 60 seconds

No Python, no dependencies, the engine is a single 600 KB binary:

# Apple Silicon Mac
curl -L https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/runner-macos-arm64.tar.gz | tar xz
curl -L -o gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf \
  https://huggingface.co/Joakimpalm-Zen/gemma-4-E2B-it-Q4_0-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf
./runner-macos-arm64 -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf -i

2.63 GB on disk, comfortable on an 8 GB Mac. Linux and Windows builds are on the same release page.

Serve it on a loopback OpenAI-compatible endpoint instead:

./runner-macos-arm64 -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve

The engine is Xyntetik Runner, a from-scratch C inference engine in public alpha. If you run this, ./runner-macos-arm64 --caps prints one JSON document describing your machine and what it admits, pasting that into an issue is the single most useful thing you can send us, and it takes ten seconds.

What this file actually is, corrected 2026-08-11

This card previously called the file "a straight Q4_0 conversion". That was wrong, and an external evaluation caught it. The true content, from the tensor table of the file itself:

F32: 283   Q4_K: 194   Q4_0: 124        general.file_type = 15 (Q4_K_M)

It was produced by requantizing a Q4_K_M GGUF conversion of google/gemma-4-E2B-it with runner --quantize --quant q4_0. The quantizer has a never-grow retention rule, and Q4_K and Q4_0 are both 4.5 bits per weight, so every Q4_K tensor was silently kept at the higher-quality K format, and only the parent's Q5/Q6/Q8 tensors actually moved to Q4_0. The result is slightly BETTER than a true Q4_0 at the same size, but it is a mixed file, and both the old filename and the old card said otherwise.

The same class of mislabeling is what the runner's own certification matrix calls the "mixed-tensor trap" when third-party quantizations do it. The same standard applies here: the file is renamed to say what it is, the bytes are UNCHANGED (hash below), and the quantizer defect that produced the stale general.file_type metadata is fixed upstream (runner commit 9a0054e: the declared type is now derived from the output histogram and the histogram is printed at quantize time).

• File: gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf (2,631,015,904 bytes)
• SHA256: 16ffaa2ea568e7a0c9b5c6ddbd39a376879f9ac007861334974de39f4cc52e3f
• Source: google/gemma-4-E2B-it via a Q4_K_M GGUF conversion
• Built with: runner --quantize --quant q4_0 (pre-9a0054e metadata behavior)

Should you use this file? Often, no, and that is worth saying plainly. Google publishes official Gemma-4 GGUFs, they are ungated and Apache 2.0, and for ordinary use you should prefer them. This artifact exists so that benchmarks quoting "gemma-4-E2B at 2.63 GB" can be reproduced against the exact bytes they were measured on, hash included. That is its whole job - which is also why the rename kept the bytes and the hash identical.

What the E-series actually is

E2B is not "Gemma 4, small". It is a distinct architecture, and two of its properties break runtimes that assume otherwise:

Per-layer embeddings (PLE). Every layer folds its own embedding table into the residual, rather than sharing one input embedding. Plus a tail of layers that own no KV of their own (shared-KV geometry). A runtime that implements "gemma-4" as a dense transformer will load this file and produce nonsense.

Per-layer FFN widths. gemma4.feed_forward_length is an array of 35 values, 6144 for the first 15 layers, 12288 for the remaining 20, because E2B sets use_double_wide_mlp. This is the interesting compatibility trap:

gemma4.feed_forward_length = [6144 x15, 12288 x20]     # not a scalar

A loader that reads that key with a scalar getter gets 0 and reports "missing model hyperparameters". Xyntetik Runner hit exactly that, and gemma-4-E4B masked it for weeks because E4B's checkpoint does not set the flag, its key is a plain scalar, so it loads fine and the missing path is never exercised. If your runtime loads E4B and fails on E2B, this is why.

Like the pruned gpt-oss artifact in this account, the file doubles as a compatibility probe: it fails loudly on runtimes that assume a scalar, and that is a useful property rather than an inconvenience.

Measured

On an 8 GB Apple Silicon M1, Xyntetik Runner 0.1.13-alpha:

2.63 GB is the number that matters on an 8 GB machine: it fits the Metal working set with room for a KV cache, where a 5 GB E4B does not.

Measured fidelity vs BF16, added 2026-08-12

This exact file (sha 16ffaa2e...) was gated against its own BF16 parent: 400 teacher-forced positions, greedy, KLD over the union of both sides' top-20 logprobs, zero-point self-check exact (0.0 KLD / 100% top-1 on the parent vs itself).

This file does not meet the fidelity bar this account now applies to new uploads. It predates that bar. The Q4_K retention (see the corrected section above) buys it roughly 20 top-1 points over a plain Q4_0 of the same model, measured at 58.0% in the same session, and it still disagrees with its parent on roughly 1 token in 4.5. What it remains good for: smoke-testing a runtime on an 8 GB machine at 20 tok/s, where nothing higher-precision fits at all. What it is not: a faithful gemma-4-E2B. If you have the memory for the official Q4_K_M (3.46 GB), prefer it; fidelity gates for the 3-5B Q4_K_M class are queued and will be published when measured. Full context: the quantization-damage-vs-model-size finding in the Xyntetik suite repo (q4_0-quality-gate-2026-08-12.md).

Running it

./runner -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve --port 8080

Any GGUF runtime with gemma-4 E-series support that reads feed_forward_length per layer will also load it.

Use the chat template. Instruction-tuned Gemma degenerates on raw untemplated prompts, the official builds do this too, so judge quality through /v1/chat/completions, not a bare completion prompt. Raw prompts are fine for speed measurement and byte-identity comparisons, where both sides degenerate equally and the comparison still holds.

Licensing

Apache 2.0. Gemma 4 ships under Apache 2.0, unlike Gemma 1-3, which carry the custom Gemma Terms of Use with their Prohibited Use Policy. This card previously said license: gemma, which was wrong: it named the older generation's terms. Corrected 2026-08-09 after checking the licence for this generation rather than the family.

If the Hub shows an imatrix tag on this repository, it is auto-derived and inaccurate, no importance matrix was used. This is a plain requantization pass (see the correction section for what "plain" turned out to mean).

Publication policy

This account publishes only artifacts that passed their quality gate; experiments that failed ship as negative-result documentation in the runner repo rather than as uploads. Corrections to published cards are made in place and dated, this card carries two (license, 2026-08-09; content labeling, 2026-08-11), because the alternative is a record that quietly stops being true.

Measured envelope

Every artifact this account publishes ships a measured-envelope sidecar (gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf.envelope.json, in this repo) — the same file xyntetik-runner reads at load: download it next to the GGUF and the runner reports the measured state at load. A measured envelope is a dated observation, not a standing quality claim — it records exactly what this file is, where it came from, and what was measured, so the claims above stay traceable.