An optimized new version of ISSAI's Kazakh base model.
Developed by Almubdi Mutaikhan with Batyr Arystanbekov (ISSAI), under the supervision of Prof. Huseyin Atakan Varol (ISSAI). See Citation.
ISSAI earlier released issai/Qwen3.5-4B-Base-Kazakh: Qwen/Qwen3.5-4B-Base with 16,000 added Kazakh tokens and continued pretraining on ~15.7B tokens of Kazakh and English (50 / 50). It made Qwen3.5 much better and much cheaper in Kazakh.
This model is its next version. It keeps the same recipe and changes two things:
Result: the best Kazakh and Russian scores of the three models, with English unchanged. The numbers are in Comparison. How it was built is in Implementation.
This is a base model: it continues text and does not follow chat instructions. Use it for fine-tuning or as the starting point for chat models. The chat version (thinking + images) built on it is issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer.
Contents: Model details · Implementation · Comparison · How to use · Limitations · Citation · Contributions
| architecture | Qwen3.5: 32 layers, hybrid (24 Gated DeltaNet linear-attention + 8 full-attention layers), hidden size 2,560, text only |
| parameters | 4.25 B |
| vocabulary | 264,077 tokens: Qwen3.5's 248,077 + 16,000 new (ids 248,077–264,076); embedding matrix padded to 264,192 rows |
| input / output embeddings | tied |
| context length used in training | 32,768 tokens |
| precision | bfloat16 |
| languages | Kazakh, Russian, English |
Qwen3.5-4B-Base
│ 1. tokenizer: +16,000 Kazakh/Russian tokens (continued BPE)
│ 2. new embeddings: mean of the old pieces (FVT)
│ 3. phase 1: train the new embedding rows only
│ 4. phase 2: train all weights on 8.4B tokens (kk / ru / en 40 / 20 / 40)
▼
Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer
Continued BPE: 16,000 new merges were learned on Kazakh and Russian text, appended after Qwen3.5's merge list. Every original token keeps its id and every original merge rule stays, so:
Example: Қазақстан Республикасының астанасы → 3 tokens (Қазақстан · Республикасының · астанасы) instead of
10 with Qwen3.5.
Each new token's embedding starts as the mean of the embeddings of the Qwen tokens it replaces (Fast Vocabulary Transfer, FVT). Input and output embeddings are tied, so this one table serves both.
Only the 16,000 new embedding rows are trained; all original weights are frozen. The new tokens learn to fit the existing model before anything else changes. 571 steps, ~16,000 sequences of 32,768 tokens, learning rate 1e-4.
One epoch over the 8.4B-token mix: 2,134 steps of 120 sequences × 32,768 tokens, learning rate 1e-5 with cosine decay to 1e-6 (2% warm-up), AdamW (β = 0.9 / 0.95), weight decay 0.1, 15 × H100, DeepSpeed ZeRO-2, ms-swift.
By tokens: 40% Kazakh, 20% Russian, 40% English; deduplicated.
The training data is not released. Each source has its own license.
Three models: the original Qwen3.5-4B-Base, ISSAI's earlier Qwen3.5-4B-Base-Kazakh, and this model.
| Qwen3.5-4B-Base | issai/Qwen3.5-4B-Base-Kazakh | this model | |
|---|---|---|---|
| added tokens | — | 16,000, Kazakh | 16,000, Kazakh + Russian |
| continued pretraining | — | ~15.7B tokens, kk / en 50 / 50 | 8.4B tokens, kk / ru / en 40 / 20 / 40 |
| Kazakh average (6 tasks) | 57.37 | 60.83 | 61.47 |
| Russian average (4 tasks) | 65.37 | 64.43 | 65.55 |
| English average (4 tasks) | 77.15 | 76.80 | 77.01 |
Tokens per word on the 488 Belebele (FLORES) passages of each language. Lower is better: the model works faster, and more text fits in its context.
| language | Qwen3.5-4B-Base | issai/Qwen3.5-4B-Base-Kazakh | this model |
|---|---|---|---|
| Kazakh | 3.48 | 1.83 (1.90× fewer) | 1.76 (1.98× fewer) |
| Russian | 1.89 | 2.23 (1.18× more) | 1.72 (1.10× fewer) |
| English | 1.25 | 1.25 | 1.25 |
Example (Russian): Бескрайняя казахская степь тянется до самого горизонта, и в тихом ауле слышно, как поёт домбра.
is 31 tokens with Qwen3.5, 34 with the earlier ISSAI tokenizer, and 29 with this one.
lm-evaluation-harness, 5-shot, full test sets, bfloat16, the same settings for all three models. Kazakh and Russian tasks are ISSAI's Kazakh/Russian versions of the benchmarks. Accuracy in % (GSM8K: exact match). ± is the standard error.
| task | Qwen3.5-4B-Base | issai/Qwen3.5-4B-Base-Kazakh | this model |
|---|---|---|---|
| MMLU (kk) | 57.49 ± 0.4 | 61.88 ± 0.4 | 61.92 ± 0.4 |
| MMLU-Pro (kk) | 35.27 ± 0.4 | 37.11 ± 0.4 | 37.43 ± 0.4 |
| MMLU-Redux (kk) | 59.07 ± 0.7 | 63.21 ± 0.6 | 64.14 ± 0.6 |
| GPQA (kk) | 31.38 ± 1.3 | 34.98 ± 1.4 | 35.99 ± 1.4 |
| ARC (kk) | 85.01 ± 0.6 | 89.80 ± 0.5 | 89.77 ± 0.5 |
| Belebele (kk) | 76.00 ± 1.4 | 78.00 ± 1.4 | 79.56 ± 1.3 |
| Kazakh average | 57.37 | 60.83 | 61.47 |
| MMLU-Pro (ru) | 43.23 ± 0.5 | 41.65 ± 0.4 | 43.24 ± 0.5 |
| GPQA (ru) | 35.15 ± 1.4 | 35.15 ± 1.4 | 37.16 ± 1.4 |
| ARC (ru) | 94.53 ± 0.4 | 93.04 ± 0.4 | 93.91 ± 0.4 |
| Belebele (ru) | 88.56 ± 1.1 | 87.89 ± 1.1 | 87.89 ± 1.1 |
| Russian average | 65.37 | 64.43 | 65.55 |
| MMLU (en) | 73.07 ± 0.4 | 74.28 ± 0.4 | 73.81 ± 0.4 |
| ARC-Challenge (en) | 61.69 ± 1.4 | 60.92 ± 1.4 | 62.46 ± 1.4 |
| GSM8K (en) | 80.74 ± 1.1 | 80.44 ± 1.1 | 79.68 ± 1.1 |
| Belebele (en) | 93.11 ± 0.8 | 91.56 ± 0.9 | 92.11 ± 0.9 |
| English average | 77.15 | 76.80 | 77.01 |
Requires transformers 5.x (tested with 5.12.1).
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="cuda")
ids = tok("Қазақстан Республикасының астанасы", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))
Apache-2.0, the same as Qwen/Qwen3.5-4B-Base.
@misc{mutaikhan2026qwen35basekazakhopt,
title = {Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer},
author = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer}}
}
model-00001-of-00002.safetensors 5.0 GBmodel-00002-of-00002.safetensors 4.9 GBQwen/Qwen3.5-4B-Base