Gemma4-12B Uncensored: 1M Context + MTP + Vision

HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (12B dense, Google QAT checkpoint) with a 1,048,576-token context baked in (4x the native 262,144), shipping with its MTP speculative-decoding draft head and vision tower. All numbers below were measured on these exact files.

Needle-in-a-haystack

The first Gemma 4 we know of certified needle-perfect at 1,048,576 tokens. Thanks to Gemma's 5:1 sliding-window attention the KV cache stays small enough that the entire 1M certification ran on a 32 GB RTX 5090 at f16 KV. One cell (786K, depth 15 percent) missed on the first seed and passed 10/10 on a second seed; both runs are in results.jsonl.

MTP speculative decoding

The draft head predicts ahead and the trunk verifies every token, so output is identical to standard decoding, only faster. Measured speedup on this uncensored trunk beats the ~35 percent claimed upstream.

Multi-hop retrieval: chains hold at 1,048,576 tokens

Beyond single-needle NIAH, we run a harder "hop" tier: chained lookups where each answer keys the next, at true 1M context. Thinking OFF, 5 chains per rung:

Perfect chain retrieval at a million tokens. (The battery also probes a multi-step variable-tracking tier; our scorer for that tier is under review before we publish it, so those numbers are held back rather than reported unverified.) There is also a stretch rung at 1,572,864 tokens (1.5M, the extended-context sibling of this model) scoring 6/10 on single-needle NIAH: past the 1M bake it degrades, exactly as honestly expected.

RULER at long context (NVIDIA's benchmark, their scorer)

NIAH proves retrieval; RULER is the harder, industry-standard suite (multi-key retrieval, variable tracking, aggregation). We ran five RULER tasks at 131K on this exact GGUF, 25 samples per task, greedy-free vendor sampling (temp 1.0, top_p 0.95, top_k 64, seed 42), scored end to end by NVIDIA's own evaluate.py (RULER commit 38da79d). The bridge script and full reproduction recipe are published in aviary-1m/tools/ruler.

All columns thinking OFF (see the headline below for why). 25 samples per task, NVIDIA's own scorer.

* The ~1M rung uses a nominal length of 917,504 with the full 1,048,576 context served. Reason, honestly: RULER sizes prompts with a reference tokenizer that undercounts this model's real tokenization by about 6 percent. Our first attempt at nominal 1,048,576 overflowed the context window and scored all-null on three tasks; that was a harness artifact, not a model failure, so we re-ran at a nominal length whose real token count (~976K) fits the served window. Zero nulls across all 125 samples in the run reported above.

The ladder climbs honestly to the top rung. Pure retrieval barely moves even at a million tokens: multiquery holds 99 and frequent-words extraction actually posts its second-best score of the whole ladder at 97.3. The cost of distance concentrates exactly where the rest of the ladder predicted: aggregation over the entire window (cwe, 5.2) and dense multi-value retrieval (57.0). Variable tracking finally bends at 68.8. This is what a real 1M model looks like: near-perfect needlework at full depth, honest degradation where the task requires holding the whole window at once. (For the record, thinking-ON at 131K averaged 45.9, half the thinking-OFF score.)

The honest headline: thinking mode halves this model's RULER score at 131K. The failure mode is specific and reproducible: on aggregation tasks (cwe/fwe) and some hard retrieval samples, the model reasons in circles and exhausts any generation budget (tested to 12K tokens) without ever emitting an answer. At temperature 0 the same samples loop deterministically; vendor sampling reduces but does not eliminate it. Those runs score 0 by RULER's rules and we report them as such, nulls and all.

Practical guidance: for long-context retrieval and aggregation work with this model, run thinking OFF. 91.8 average at 131K with zero unanswered samples is the model's real capability; thinking mode is the liability, not the context window. Raw prediction files, both conditions, are in the repo history for anyone who wants to re-score.

Files

Every file, every mirror

Nothing was discontinued: every quant is one click away. Hugging Face carries the curated picks, ModelScope always carries everything, and Ollama serves ready-to-run tags.

On Ollama every tag ships with the vision tower bundled and the 1M rope metadata baked in.

Run it

llama.cpp, everything on:

llama-server -m gemma4-12b-uncensored-1M-Q4.gguf \
  -c 1048576 -np 1 --jinja \
  -md mtp-gemma-12b.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --mmproj mmproj-gemma12b-hauhau.gguf

Ollama (1M and vision work; Ollama has no speculative decoding yet, so the MTP head adds no speed there):

FROM ./gemma4-12b-uncensored-1M-Q4.gguf
RENDERER gemma4
PARSER gemma4
PARAMETER num_ctx 262144

The RENDERER and PARSER lines avoid imported-GGUF template bugs under tool-heavy use. Raise num_ctx as memory allows.

How to actually use a 1M-context model

Long context is a capability, not a magic mode. Habits that measurably help (from our own RULER and hop testing on this model):

1. Re-state your standing instructions near the end of long prompts. Recency beats depth.
2. Prefer one big reference dump over a long accumulated chat. Fresh session per task, context used as a library.
3. After any compaction or summarization, repeat your active rules yourself.
4. Run thinking OFF for retrieval and aggregation work. The RULER table above is unambiguous: thinking mode halves this model's score at long range.
5. Expect the extremes to cost time: prefill at 500K+ is slow on any hardware. Budget for it.

How this was built

YaRN rope-scaling metadata (factor 4.0 over native 262,144) baked into the GGUF header with gguf-py; weights are bit-identical to the HauhauCS release, no fine-tuning. Gemma 4's dual-rope design takes YaRN on its global-attention layers. Certification harness: 10 needles per rung at depths 5 to 95 percent, temperature 0, seeded prompts, f16 KV only. Method and tooling: github.com/satindergrewal/aviary-1m.

For base capability benchmarks see Google's official Gemma 4 cards; uncensoring quality versus the official trunk has not been independently benchmarked here.

Credits

Base model and QAT: Google (Gemma license; its terms flow down to these files). Uncensoring and packaging: HauhauCS. MTP head: Unsloth (via the HauhauCS repo). 1M YaRN extension, benchmarking, and certification: SatGeze.

Sister repos: 12B | 26B-A4B | 31B | Qwen3.6-35B

Mirrors: Hugging Face | ModelScope