gemma-4-E2B-it-enja

google/gemma-4-E2B-it with its vocabulary pruned to English and Japanese (conservative variant): 262,144 → 164,224 tokens (−37%). The model weights are otherwise unchanged; no retraining was done.

Gemma 4 E2B has a 262,144 × 8,960 per-layer embedding table in addition to its 262,144 × 1,536 input/output embedding, so every dropped token saves 10,496 parameters. That makes the vocabulary roughly half of the text model.

See also brishen/gemma-4-E2B-it-enja-aggr, a more aggressive variant (131,648 tokens) that also drops ASCII words unseen in English/Japanese/code corpora.

What was removed

• Tokens containing scripts other than Latin, kana and Japanese kanji (Cyrillic, Hangul, Arabic, Thai, Devanagari, simplified-only Han, ...), plus the <unusedN> placeholders.
• Multi-character Han words and accented Latin words that do not occur in English/Japanese corpora (Japanese Wikipedia, KFTT, ParallelFiction, wikitext-103, FineWeb-Edu).
• In this variant, every ASCII token is kept, so code, URLs and names tokenize exactly as before.

Always kept: special and control tokens (chat, tool, image, audio), all 256 <0xNN> byte-fallback tokens, kana, single kanji in the cp932 repertoire, punctuation, symbols and emoji.

The keep set is closed under BPE merges. As a result, text whose original tokenization used only kept tokens tokenizes identically, with ids remapped, and on kept tokens the logits are bit-identical to the original model's. Text in removed scripts still round-trips through byte fallback, but the model becomes much worse at it.

prune_manifest.json lists keep_ids: entry i is the original token id of new token i.

Evaluation

Zero-shot, greedy decoding. In parentheses: the share of examples where this model's prediction or output is identical to the original's.

None of the differences is statistically significant (95% paired-bootstrap CIs include 0). Out of scope, as intended: zh/ko/ru bits-per-byte ×2–4 (byte fallback); de/fr/es +10–14%.

Usage

Use it exactly like the base model; the chat template and processor config are unchanged.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("brishen/gemma-4-E2B-it-enja")
model = AutoModelForCausalLM.from_pretrained("brishen/gemma-4-E2B-it-enja", dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "日本の首都はどこですか？"}]
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• Only text was evaluated. The vision and audio towers are included unchanged, and their placeholder token ids were remapped in config.json, but image and audio inputs have not been tested.
• Languages other than English and Japanese degrade sharply.
• The original's <unusedN> tokens are gone. To add new special tokens, prune with --keep-unused or resize the embeddings.

Reproduce

Made with the pruning/eval code in gemma-4-pruned: uv run gemma-prune --out models/gemma-4-E2B-it-enja.

License

Apache 2.0, inherited from Gemma 4. See the Gemma 4 license. Original model by Google DeepMind.