gemma-4-E2B-it-enja-aggr

google/gemma-4-E2B-it with its vocabulary pruned to English and Japanese (aggressive variant): 262,144 → 131,648 tokens (−50%). The model weights are otherwise unchanged; no retraining was done.

Gemma 4 E2B has a 262,144 × 8,960 per-layer embedding table in addition to its 262,144 × 1,536 input/output embedding, so every dropped token saves 10,496 parameters. That makes the vocabulary roughly half of the text model.

See also brishen/gemma-4-E2B-it-enja, the conservative variant (164,224 tokens), which keeps every ASCII token and is the safer default.

What was removed

• Tokens containing scripts other than Latin, kana and Japanese kanji (Cyrillic, Hangul, Arabic, Thai, Devanagari, simplified-only Han, ...), plus the <unusedN> placeholders.
• Multi-character Han words and accented Latin words that do not occur in English/Japanese corpora (Japanese Wikipedia, KFTT, ParallelFiction, wikitext-103, FineWeb-Edu).
• In this variant, multi-letter ASCII tokens are also kept only if seen in English, Japanese or code corpora (The Stack smol-xl, 24 languages). That removes diacritic-free words of other Latin-script languages (Indonesian, Dutch, ...), but rare English words and identifiers now split into smaller pieces.

Always kept: special and control tokens (chat, tool, image, audio), all 256 <0xNN> byte-fallback tokens, kana, single kanji in the cp932 repertoire, punctuation, symbols and emoji.

The keep set is closed under BPE merges. As a result, text whose original tokenization used only kept tokens tokenizes identically, with ids remapped, and on kept tokens the logits are bit-identical to the original model's. Text in removed scripts still round-trips through byte fallback, but the model becomes much worse at it.

prune_manifest.json lists keep_ids: entry i is the original token id of new token i.

Evaluation

Zero-shot, greedy decoding. In parentheses: the share of examples where this model's prediction or output is identical to the original's.

None of the differences is statistically significant (95% paired-bootstrap CIs include 0). Out of scope, as intended: zh/ko/ru bits-per-byte ×2–4 (byte fallback); de/fr/es +35–39%.

Usage

Use it exactly like the base model; the chat template and processor config are unchanged.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("brishen/gemma-4-E2B-it-enja-aggr")
model = AutoModelForCausalLM.from_pretrained("brishen/gemma-4-E2B-it-enja-aggr", dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "日本の首都はどこですか？"}]
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• Only text was evaluated. The vision and audio towers are included unchanged, and their placeholder token ids were remapped in config.json, but image and audio inputs have not been tested.
• Languages other than English and Japanese degrade sharply.
• The original's <unusedN> tokens are gone. To add new special tokens, prune with --keep-unused or resize the embeddings.

Reproduce

Made with the pruning/eval code in gemma-4-pruned: uv run gemma-prune --ascii-policy evidence --out models/gemma-4-E2B-it-enja-aggr.

License

Apache 2.0, inherited from Gemma 4. See the Gemma 4 license. Original model by Google DeepMind.