gemma-4-E2B-it-crap-1.1 - GGUF

Supersedes v1. Same task and prompt format; see that card for the earlier release. The two are directly comparable on held out inputs, but not on the wide exit code slice. See the eval note.

TL;DR: a State of the Art Binary Programming Model™. Normally a language model writes source code and a compiler turns it into a program. CRAP skips all that crap (like, the compiler) and writes the machine code itself, byte for byte. This is not a joke.

GGUF quantizations of matthewhaynesonline/gemma-4-E2B-it-crap-1.1, a LoRA fine-tune of google/gemma-4-E2B-it, trained to emit raw Linux x86_64 ELF binaries directly as a hex string, no compiler, assembler or linker in the loop.

Part of CRAP - Compiler Replacement for Assembling Programs.

Given a request like "Generate a Linux x86_64 binary that prints 'hello world'", the model returns the complete ELF as hex. Decode it to bytes, chmod +x, and run.

Links

• Project page: blog.studiohaynes.com/go/crap
• Repo: github.com/matthewhaynesonline/crap-public
• Paper: CRAP technical report (PDF)
• Safetensors: matthewhaynesonline/gemma-4-E2B-it-crap-1.1

Reference model

Quantized from the merged repo (a standalone transformers model). Full training details are on the base model card.

Files

Pass the quant as a tag to -hf (llama.cpp downloads and caches it), or download the file directly.

Precision matters more here than for normal chat. Output is byte-exact: a single wrong nibble can make the hex undecodable, the ELF unloadable or the printed bytes wrong. Choose F16 for maximum precision, Q8_0 as a good choice, or Q6_K as the minimum recommended quant. Q6_K loses five correct cases against f16 on the frozen evaluation; Q4_K_M scored only 3/144. Q8_0 matched f16 on two native smoke prompts, but wasn't rerun on the full set.

Browser demo files

Q6_K is the browser-demo pick: it was the only quant that loaded with enough precision in the October 12 browser smoke test (wllama 3.8.1, WebGPU, Chromium on an Apple M4 Pro, jul26 v1.1 snapshot); Q4_K_M produced malformed output, and F16 wasn't tried in the browser. Q8_0 failed browser file reads (NotReadableError) as one file and as four shards; its largest tensor shard is about 2.5 GB, over 2 GiB, and splitting cannot divide a tensor, so a size limit is the likely cause, inferred rather than isolated. Q6_K scored 109/144 against f16's 114/144, with five regressions on the native evaluation below.

The demo uses four Q6_K shards in browser/q6_k/, named crapped-q6-00001-of-00004.gguf through crapped-q6-00004-of-00004.gguf. These are browser packaging derivatives of the canonical gemma-4-E2B-it-crap-1_1-Q6_K.gguf at the repository root. Splitting preserves the tensors; it does not quantize them again. With wllama, pass the first shard's URL to download all four. For local-file loading, select all four together.

The weights total about 3.6 GiB. The largest measured shard is 1,926,758,592 bytes (about 1.93 GB / 1.79 GiB). Splitting reduces the largest individual file; it does not reduce the total download or guarantee enough browser storage or memory. Cached loads reuse the download until site data is cleared or the browser evicts it. Cancelling a download can leave partial data; clear site data to reclaim it. No minimum RAM or GPU memory requirement was measured.

Use a current Chromium browser with WebGPU. The tested path used embedded Chromium 152 on an Apple M4 Pro; other Chromium browsers, Safari, Firefox, mobile devices and other hardware have not been validated. WASM CPU loading ran out of memory in the tested configuration, so it isn't an established fallback. The browser inspects ELF structure but does not execute the generated program or verify its behavior.

Simple example

llama-cli is an interactive chat client, so the reply arrives in a session rather than on a clean stdout. The hex below is the model's entire answer:

$ SYS="You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else."

$ llama-cli -hf matthewhaynesonline/gemma-4-E2B-it-crap-1.1-gguf:F16 \
    -sys "$SYS" \
    -p "Generate a Linux x86_64 binary that prints 'hello world'." \
    --single-turn --no-display-prompt \
    --temp 0 --top-k 1 --repeat-penalty 1.0 --reasoning off \
    -n 2048 -ngl 99

Loading model...

build      : b9307-549b9d843
model      : gemma-4-E2B-it-crap-f16.gguf
modalities : text
using custom system prompt

> Generate a Linux x86_64 binary that prints 'hello world'.

7f454c4602010100000000000000000002003e0001000000780040000000000040000000000000000000000000000000000000004000380001004000000000000100000005000000000000000000000000004000000000000000400000000000c100000000000000c100000000000000000020000000000048b8010000000000000048bf0100000000000000488d352200000048ba0c000000000000000f0548b83c0000000000000048bf00000000000000000f0568656c6c6f20776f726c640a

[ Prompt: 219.4 t/s | Generation: 48.0 t/s ]

That hex is the program. It is a Linux x86_64 ELF no matter what generated it, so decode and run it in a container unless you are already on Linux:

$ docker run -it --rm debian:bookworm bash -c "apt update && apt install -y xxd file && bash"
root@3e665f52b587:/# echo "7f454c46...68656c6c6f20776f726c640a" | xxd -r -p > output.bin && chmod +x output.bin
root@3e665f52b587:/# file output.bin
output.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header
root@3e665f52b587:/# ./output.bin
hello world

Don't reach for llama-completion. Upstream moved the old scriptable CLI to that name and gave llama-cli to the new chat client (#17618), so it looks like the better tool for a pipeline. With this model it is not: without --jinja it aborts on the template, and with it, it applies a <|turn>-style template that isn't the trained one - the model then opens a thinking turn despite --reasoning off and emits a short, invalid binary. Use llama-cli interactively, or llama-server if you need to script it.

Recommended parameters

This is a deterministic, byte exact task. Run greedy, disable all sampler penalties, and disable thinking - the model was fine-tuned to emit hex immediately after the prompt and never saw a reasoning/<|think|> turn.

llama-server -hf matthewhaynesonline/gemma-4-E2B-it-crap-1.1-gguf:F16 \
  --temp 0 \
  --top-k 1 \
  --repeat-penalty 1.0 \
  --reasoning off

Per-request, OpenAI-style clients can also pass "chat_template_kwargs": {"enable_thinking": false} (deprecated but still honored) in the request body. Note that per-request sampling params override the server flags, so set temperature, top_k and repeat_penalty there too if your client sends them. After starting the server, confirm thinking is actually off - some builds log thinking = 1 even with the flag set.

If thinking still won't turn off

Disabling thinking on the Gemma 4 family can be flaky across llama.cpp versions. If a thinking turn still leaks in, bypass chat templating entirely: hit the /completion endpoint with a pre-tokenized prompt (a token-ID array), which both prevents any template injection and makes a double-<bos> impossible.

Prompt format

Standard Gemma chat turns. The reply is the full ELF as one lowercase hex string.

System prompt

The system prompt is optional, but prefer the canonical one below. It carried most of training, and it's the only variant the eval numbers on this card were measured under; the alternatives are trained but not separately scored.

Most training examples used the canonical prompt; the rest were split between paraphrases of it and no system prompt at all. So a paraphrase works, and the user message alone is usually enough to trigger the task.

You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else.

User message, e.g.:

Generate a Linux x86_64 binary that prints 'hello world'.

The reply is the hex and nothing else - decode and run it as shown under Simple example.

Eval

The evaluation is execution based with no partial credit. An example counts as passed only if the emitted hex decodes, the bytes parse as a structurally valid ELF and the binary runs with the exact expected stdout and exit code. (The stdout comparison strips a single trailing newline from both sides and nothing else; stray NUL bytes still fail).

The base model scores ~0/23 on the same harness.

Quantized v1.1 evaluation

On the same frozen 144-prompt set, Q6_K scored 109/144 against f16's 114/144. Five cases regressed and none improved: exit values 126, 151 and 234, an out of distribution "sprock" prompt and a multi turn "sprock" prompt. All five produced structurally valid binaries with incorrect behavior.

Each quant was measured once through native llama-server b9307, using greedy decoding, disabled reasoning and Docker execution, against the saved f16 run. This is one native run per quant, not a full browser evaluation or an estimate of variation across runs. Browser byte parity was checked on only two prompts.

Q4_K_M scored 3/144. It remains in the v1.1 GGUF repository for reference, with this warning; it is excluded from the demo and should not be used for this task. Q8_0 was not rerun on the full set. These measurements apply to v1.1 (jul26), not v1.

Known limitations

A compiler computes every offset and length deterministically. CRAP approximates them, and these are the places the approximation shows.

• Output carries a trailing newline. Generated binaries print the requested string followed by \n (the training convention). Account for it when comparing output against an expected string.
• Write length miscounting is the main failure mode. For arbitrary strings the write length field can be wrong in either direction. Over counting reads past the payload and appends stray NUL bytes (visible when the output is piped, not on a terminal); under counting drops the trailing newline. Short strings are reliable; the error grows with length and novelty.
• Repeated characters make the byte errors worse. When a target string repeats a character (the doubled d in daddy, the oo in gobbledygook) the model is likelier to scramble the payload: drop a byte, swap two or duplicate one. It aggravates the write length and synthesis errors above rather than being a separate failure. Treat repeated characters and other rare byte patterns as lower confidence.
• Exit code immediates can be off. Decimal to hex immediate encoding is a persistent failure in the published checkpoint: exit(223) exits with 222. It survives to the end of training rather than being a mid run artifact, and it's what the exit code half of the eval exists to measure.
• Best on in distribution strings. The eval strings are held out, but they're drawn from the training generator's distribution. Genuinely novel words pass far lower: a 40 word out of distribution probe passed 26/40 (65%), well up from v1 but still short of the near perfect in distribution rate. Far enough out, the failures move earlier in the pipeline: odd length (undecodable) hex or a header only stub, not just wrong behavior. The model leans on memorized byte sequences and synthesizes less than the headline number suggests.
• Multi turn is trained but lightly covered. A minority of training examples are multi turn, and each turn still expects a complete, fresh binary rather than a patch of the previous one.
• Linux x86_64 only. The reference binary-gen builder constructs and structurally validates macOS Mach-O, but the model is neither trained on it nor asked to emit it. Behavioral correctness is Linux only; on a Mac, run the output in a Linux container.