Qwen2.5-0.5B with a Russian-pruned vocabulary
Qwen/Qwen2.5-0.5B with its vocabulary cut down to the tokens needed for Russian text, plus the BPE-merge closure that keeps tokenization consistent. The embedding matrix (tied with the output head) is sliced to match, and the transformer weights are unchanged.
| Original | Pruned | |
|---|---|---|
| Vocabulary | 151,936 | 11,904 |
| Parameters | 494M | 369M |
| Held-out bits per byte (920 Russian paragraphs) | 0.7208 | 0.7206 |
Language-modelling quality on Russian is unchanged, with about 25% fewer parameters.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "businesslion/qwen2.5-0.5b-ru-pruned"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)