Gemma 4 31B IT with attention block 48 removed (Q4_0, GGUF)

Research artifact, not a general-purpose release.

• What was changed: attention sublayer of block 48 (of 60) physically removed with --remove-sublayer; declared in the header, weights exactly zero.
• Why it exists: to measure sublayer removal on a real file: the cut frees 64 MiB of KV cache at 4k context and 512 MiB at 32k, plus 74.3 MB of file.
• Parent model: google/gemma-4-31B-it (Google's QAT Q4_0 release)
• Useful for normal inference? not as a smaller or faster daily driver; it tracks its parent at clean KLD 0.0223 over 44,413 positions, but it exists for people measuring sublayer removal.
• Experiment: Model Surgery and Scale Recovery
• Evidence: the ablation figures on this card
• Runner compatibility: loads only in Runner 0.4.7 or later, CPU path (--gpu off); llama.cpp refuses it by name.

This file frees 64 MiB of KV cache at 4k context and 512 MiB at 32k, and 74.3 MB of file, by physically removing one attention sublayer, block 48 of 60, that a complete 180-row ablation map of this model measured as the cheapest dispensable cut. It matches its parent to clean mean KLD 0.0223 with margin-qualified top-1 98.72% over 44,413 held-out positions (measured 2026-08-30 on the zeroed-weights form of the same cut, carried to this file by bit identity on 2026-09-04), and the runner's raw-protocol bar on the Q4_0 pair reads KLD 0.0239, margin-qualified top-1 99.25% (measured 2026-09-04). The KV figure is the point: it grows with context, while the disk saving is 0.42% of the file.

Serve with xyntetik-runner — tool calls that still parse when the budget runs out

This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:

When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.

Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models — tight context, slow generation — that is the difference between an agent loop that finishes and one that retries from scratch.

Measured — what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1→64):

Every engine completes at 64 tokens — the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. † TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.

Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) — it guarantees the shape of a call at any quantization, not its contents.

Quickstart

# build the single binary (no deps), then serve this GGUF:
runner -m gemma-4-31B-it-attn48-removed-q4_0.gguf --gpu off --serve
# point any OpenAI-compatible client at http://localhost:8080/v1

Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.

Read this first. This is a research artifact for people measuring sublayer removal, not a smaller or faster daily driver. It loads only in xyntetik-runner v0.4.7 or later (released 2026-09-04). Runner v0.4.6 and earlier, and stock llama.cpp (measured on b10076), refuse it with missing tensor 'blk.48.attn_q.weight'. That is the intended failure: the absence is declared in the header, never hidden. Today only the runner's CPU path omits the removed block, so pass --gpu off; the device paths are open work.

Run it

The file is 17.6 GB and runs on the CPU path, so it wants a machine with roughly 20 GB of free memory and many cores. On a laptop this is a measurement subject, not a chat model.

# a release binary (Linux x86-64 shown; macOS arm64 and Windows are on the same page)
curl -L https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/download/v0.4.7/runner-linux-x86_64.tar.gz | tar xz
curl -L -o gemma-4-31B-it-attn48-removed-q4_0.gguf \
  https://huggingface.co/Joakimpalm-Zen/gemma-4-31B-it-attn48-removed-Q4_0-GGUF/resolve/main/gemma-4-31B-it-attn48-removed-q4_0.gguf
./runner-linux-x86_64 -m gemma-4-31B-it-attn48-removed-q4_0.gguf --gpu off -v -p "hi" -n 32

The -v banner prints sublayers removed attn:48 and a kv cache line 67.1 MB smaller than the parent's at the default 4,096 context.

What was removed, exactly

Block 48 keeps its pre-attention norm, post-attention norm and its whole FFN. The absence is declared the way llama.cpp's own per-layer arrays can declare it: gemma4.attention.head_count and head_count_kv are per-block arrays with a 0 at block 48 (32 and 16 elsewhere), the reading llama.cpp's Nemotron-51B "deci" graph uses for attention-free blocks. The other 827 tensors are byte-identical to the parent.

• File: gemma-4-31B-it-attn48-removed-q4_0.gguf
• SHA256: d7eaef9a8c05d8ad5cc63e4b37767113498b6ec6a31863519f5399f12cdb3134
• Parent: google/gemma-4-31B-it-qat-q4_0-gguf, file gemma-4-31B_q4_0-it.gguf, revision 59dde245, SHA256 179cfb99212709597eae5929112cfca677e1bbf566178b479ae1da0c4772874b (Google's own QAT Q4_0 release of google/gemma-4-31B-it)

Why block 48

A complete ablation map of the 31B decoder (60 blocks, whole-block, MLP-only and attention-only cuts, 180 rows) was scored against the frozen BF16 parent on a held-out split with the house bar of clean mean KLD at or under 0.05 and margin-qualified top-1 at or above 97%. Five attention-only cuts passed, all in the late band; attn:48 was the cheapest at KLD 0.0223. Cuts do not combine (every pair fails the bar), so the deployable figure is one cut, and this is it. The masked, zeroed-weights and removed forms of the cut are the same function: the zeroed and removed files score bit-identically over 4,562 positions (maximum absolute logprob difference 0.0), and the zeroed file scored bit-identically to the map's mask row at float64.

Measured envelope

Limits, read before quoting

• Parent-agreement is not a capability benchmark. Every number above says how closely this file tracks its own parent on a held-out corpus, not how well either model performs on a task. This project has published a case where a model cleared the same bar and still failed a paired benchmark test.
• "No quality loss" is not claimed and is not supported by this evidence. The artifact disagrees with its parent on about 3.5% of held-out positions.
• Runner-only, CPU-only, for now. Any other runtime refuses the file by name. The runner's GPU paths refuse it too until they learn to skip a removed block.
• The disk saving is the small end of this program's results (0.42% of the file). The reason to ship the file is the KV cache, which is 90% of the disk saving at 4k context and seven times it at 32k.
• Two fidelity rows, two protocols. The BF16 house number comes from the research scorer on a decontaminated held-out split; the Q4_0 row comes from the runner's raw-completions script on its mixed fixture corpus at 400 positions. They are quoted beside each other, not as each other.

Reproduce

# the parent, from Google's repo
curl -L -o gemma-4-31B_q4_0-it.gguf \
  https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/resolve/main/gemma-4-31B_q4_0-it.gguf
sha256sum gemma-4-31B_q4_0-it.gguf   # 179cfb99...

# the cut; survivors are copied byte for byte, so the output hash reproduces
./runner -m gemma-4-31B_q4_0-it.gguf --quantize gemma-4-31B-it-attn48-removed-q4_0.gguf \
  --remove-sublayer attn:48
sha256sum gemma-4-31B-it-attn48-removed-q4_0.gguf   # d7eaef9a...

# the Q4_0-pair fidelity row
python3 scripts/kld-compare-raw.py --model-a gemma-4-31B_q4_0-it.gguf \
  --model-b gemma-4-31B-it-attn48-removed-q4_0.gguf --runner ./runner \
  --corpus tests/fixtures/mixed-corpus.txt --max-positions 400 --out kld.json

Provenance

SURGERY.json beside the file records the parent revision and hash, the exact tensors removed, the byte counts, the header declaration, the recipe and the measurements that justified the cut. The .envelope.json sidecar is the runner-readable measured envelope. The removal mechanism, its gates and the measurements on this file are documented in the runner repository under docs/sublayer-removal.md.