gemma-4-E2B-it-crap - GGUF

A newer version is available: v1.1 supersedes this model, with stronger robustness on out of distribution strings, repeated characters and multi turn prompts. This card documents the original release.

TL;DR: a State of the Art Binary Programming Model™. Normally a language model writes source code and a compiler turns it into a program. CRAP skips all that crap (like, the compiler) and writes the machine code itself, byte for byte. This is not a joke.

GGUF quantizations of matthewhaynesonline/gemma-4-E2B-it-crap, a LoRA fine-tune of google/gemma-4-E2B-it, trained to emit raw Linux x86_64 ELF binaries directly as a hex string, no compiler, assembler or linker in the loop.

Part of CRAP - Compiler Replacement for Assembling Programs.

Given a request like "Generate a Linux x86_64 binary that prints 'hello world'", the model returns the complete ELF as hex. Decode it to bytes, chmod +x, and run.

Links

• Project page: blog.studiohaynes.com/go/crap
• Repo: github.com/matthewhaynesonline/crap-public
• Paper: CRAP technical report (PDF)
• Safetensors: matthewhaynesonline/gemma-4-E2B-it-crap

Reference model

Quantized from the merged repo (a standalone transformers model). Full training details are on the base model card.

Files

Pass the quant as a tag to -hf (llama.cpp downloads and caches it), or download the file directly.

Precision matters more here than for normal chat. Output is byte-exact: a single wrong nibble can make the hex undecodable, the ELF unloadable or the printed bytes wrong. Choose F16 for maximum precision; Q8_0 is likely a good choice and Q6_K likely the minimum usable quant. No v1 quant was measured: these tiers extrapolate from v1.1's measurements, and precision risk rises as quants shrink. Measure your chosen quant on your own eval.

Simple example

llama-cli is an interactive chat client, so the reply arrives in a session rather than on a clean stdout. The hex below is the model's entire answer:

$ SYS="You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else."

$ llama-cli -hf matthewhaynesonline/gemma-4-E2B-it-crap-gguf:F16 \
    -sys "$SYS" \
    -p "Generate a Linux x86_64 binary that prints 'hello world'." \
    --single-turn --no-display-prompt \
    --temp 0 --top-k 1 --repeat-penalty 1.0 --reasoning off \
    -n 2048 -ngl 99

Loading model...

build      : b9307-549b9d843
model      : gemma-4-E2B-it-crap-f16.gguf
modalities : text
using custom system prompt

> Generate a Linux x86_64 binary that prints 'hello world'.

7f454c4602010100000000000000000002003e0001000000780040000000000040000000000000000000000000000000000000004000380001004000000000000100000005000000000000000000000000004000000000000000400000000000c100000000000000c100000000000000000020000000000048b8010000000000000048bf0100000000000000488d352200000048ba0c000000000000000f0548b83c0000000000000048bf00000000000000000f0568656c6c6f20776f726c640a

[ Prompt: 219.4 t/s | Generation: 48.0 t/s ]

That hex is the program. It is a Linux x86_64 ELF no matter what generated it, so decode and run it in a container unless you are already on Linux:

$ docker run -it --rm debian:bookworm bash -c "apt update && apt install -y xxd file && bash"
root@3e665f52b587:/# echo "7f454c46...68656c6c6f20776f726c640a" | xxd -r -p > output.bin && chmod +x output.bin
root@3e665f52b587:/# file output.bin
output.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header
root@3e665f52b587:/# ./output.bin
hello world

Don't reach for llama-completion. Upstream moved the old scriptable CLI to that name and gave llama-cli to the new chat client (#17618), so it looks like the better tool for a pipeline. With this model it is not: without --jinja it aborts on the template, and with it, it applies a <|turn>-style template that isn't the trained one - the model then opens a thinking turn despite --reasoning off and emits a short, invalid binary. Use llama-cli interactively, or llama-server if you need to script it.

Recommended parameters

This is a deterministic, byte exact task. Run greedy, disable all sampler penalties, and disable thinking - the model was fine-tuned to emit hex immediately after the prompt and never saw a reasoning/<|think|> turn.

llama-server -hf matthewhaynesonline/gemma-4-E2B-it-crap-gguf:F16 \
  --temp 0 \
  --top-k 1 \
  --repeat-penalty 1.0 \
  --reasoning off

Per-request, OpenAI-style clients can also pass "chat_template_kwargs": {"enable_thinking": false} (deprecated but still honored) in the request body. Note that per-request sampling params override the server flags, so set temperature, top_k and repeat_penalty there too if your client sends them. After starting the server, confirm thinking is actually off - some builds log thinking = 1 even with the flag set.

If thinking still won't turn off

Disabling thinking on the Gemma 4 family can be flaky across llama.cpp versions. If a thinking turn still leaks in, bypass chat templating entirely: hit the /completion endpoint with a pre-tokenized prompt (a token-ID array), which both prevents any template injection and makes a double-<bos> impossible.

Prompt format

Standard Gemma chat turns. The reply is the full ELF as one lowercase hex string.

System prompt

The system prompt is optional, but prefer the canonical one below. It carried most of training, and it's the only variant the eval numbers on this card were measured under; the alternatives are trained but not separately scored.

Most training examples used the canonical prompt; the rest were split between paraphrases of it and no system prompt at all. So a paraphrase works, and the user message alone is usually enough to trigger the task.

You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else.

User message, e.g.:

Generate a Linux x86_64 binary that prints 'hello world'.

The reply is the hex and nothing else - decode and run it as shown under Simple example.

Eval

The evaluation is execution based with no partial credit. An example counts as passed only if the emitted hex decodes, the bytes parse as a structurally valid ELF and the binary runs with the exact expected stdout and exit code. (The stdout comparison strips a single trailing newline from both sides and nothing else; stray NUL bytes still fail).

The base model scores ~0/23 on the same harness.

That figure is for the merged reference model, not these GGUF files: the quantizations here are not separately scored. F16 was byte-identical to the reference on the five-string parity; lower quants may not track. Treat it as an upper bound and measure on your own eval before relying on Q6_K / Q4_K_M. With n=23, one example is 4.3 percentage points.

Known limitations

A compiler computes every offset and length deterministically. CRAP approximates them, and these are the places the approximation shows.

• Output carries a trailing newline. Generated binaries print the requested string followed by \n (the training convention). Account for it when comparing output against an expected string.
• Write length miscounting is the main failure mode. For arbitrary strings the write length field can be wrong in either direction. Over counting reads past the payload and appends stray NUL bytes (visible when the output is piped, not on a terminal); under counting drops the trailing newline. Short strings are reliable; the error grows with length and novelty.
• Repeated characters make the byte errors worse. When a target string repeats a character (the doubled d in daddy, the oo in gobbledygook) the model is likelier to scramble the payload: drop a byte, swap two or duplicate one. It aggravates the write length and synthesis errors above rather than being a separate failure. Treat repeated characters and other rare byte patterns as lower confidence.
• Exit code immediates can be off. Decimal to hex immediate encoding is a persistent failure in the published checkpoint: exit(223) exits with 222. It survives to the end of training rather than being a mid run artifact, and it's what the exit code half of the eval exists to measure.
• Best on in distribution strings. The eval strings are held out, but they're drawn from the training generator's distribution. Genuinely novel words pass far lower: a 12 word out of distribution probe passed 4/12 (33%), and at n=12 that figure carries roughly 14 points of standard error. Far enough out, the failures move earlier in the pipeline: odd length (undecodable) hex or a header only stub, not just wrong behavior. The model leans on memorized byte sequences and synthesizes less than the headline number suggests.
• Single turn only. Trained on single turn examples; multi turn conversations degrade. Treat each request as independent.
• Linux x86_64 only. The reference binary-gen builder constructs and structurally validates macOS Mach-O, but the model is neither trained on it nor asked to emit it. Behavioral correctness is Linux only; on a Mac, run the output in a Linux container.