InDEXt-125M
Bilingual (English & Russian) 125M Small Language Model — Trained from Scratch
Overview
InDEXt-125M is a compact, bilingual (English and Russian) causal Small Language Model (SLM) trained completely from scratch on 2.48B tokens following Chinchilla scaling laws (~20 tokens per parameter).
Key design highlights:
- Custom Bilingual Tokenizer: ByteLevel BPE (
vocab_size = 32,576) trained from scratch on a balanced EN/RU corpus for efficient Cyrillic & Latin tokenization. - Modern Llama-3 Style Decoder: Built in pure PyTorch with
RoPE,RMSNorm,SwiGLU,Weight Tying, andFlashAttention-2(via SDPA). - Two-Stage Pipeline: Chinchilla pretraining on curated EN/RU web & educational corpora followed by a dedicated 100% bilingual dialogue SFT polish stage.
Architecture & Specifications
| Component | Specification |
|---|---|
| Model Type | LlamaForCausalLM (Decoder-Only Causal LM) |
| Total Parameters | 124.13M (124,128,000 exact, weight-tied) |
| Backbone / Brain | 99.11M (12 layers, 12 heads, head_dim = 64, d_model = 768, hidden_dim = 2560) |
| Vocab / Embeddings | 25.02M (Custom EN/RU ByteLevel BPE, vocab_size = 32,576) |
| Positional Encoding | Rotary Position Embeddings (RoPE, theta = 10000.0) |
| Normalization | Pre-norm RMSNorm (eps = 1e-5) |
| MLP Activation | SwiGLU |
| Attention | Causal Multi-Head Attention (FlashAttention-2 / SDPA) |
| Context Length | 1024 tokens |
| Precision | bfloat16 |
| Weight Format | SafeTensors (model.safetensors) |
Training Procedure
| Hyperparameter | Value |
|---|---|
| Methodology | Chinchilla-Optimal (~20 tokens / parameter) |
| Training Budget | 2.48B tokens (5,045 steps) |
| Global Batch Size | 491,520 tokens/step (batch_size = 96, grad_accum_steps = 5, block_size = 1024) |
| Optimizer | Fused AdamW (betas = (0.9, 0.95), weight_decay = 0.1, grad_clip = 1.0) |
| LR Schedule | Cosine decay (6e-4 → 6e-5) with 250 warmup steps |
Stage 1 (Steps 1 – 4400) |
Bilingual Pretraining (EN + RU + Dialogues) |
Stage 2 (Steps 4401 – 5045) |
100% Bilingual Dialogue SFT Polish (EN + RU 50/50) |
Training Datasets
| Dataset | Language | Role | Pretrain Weight |
|---|---|---|---|
HuggingFaceFW/fineweb-edu (sample-10BT) |
English | Educational Web Pretrain | 0.35 |
HuggingFaceFW/fineweb-2 (rus_Cyrl) |
Russian | Russian Web Pretrain | 0.35 |
HuggingFaceTB/smoltalk (all) |
English | Dialogue & Instruction | 0.15 (0.50 in SFT) |
Vikhrmodels/GrandMaster-PRO-MAX |
Russian | Dialogue & Instruction | 0.15 (0.50 in SFT) |
Prompt Format
Use explicit role tags with newline separation for chat inference:
<|user|>
Your question here
<|assistant|>
Special Tokens: <|endoftext|>, <|pad|>, <|user|>, <|assistant|>
Inference (SafeTensors)
import json
import torch
from safetensors.torch import load_file
from transformers import PreTrainedTokenizerFast
model_dir = "./InDEXt-125M"
with open(f"{model_dir}/config.json", "r", encoding="utf-8") as f:
cfg = json.load(f)
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_dir)
model = InDEXtForCausalLM(cfg).to("cuda", dtype=torch.bfloat16)
state_dict = load_file(f"{model_dir}/model.safetensors", device="cuda")
model.load_state_dict(state_dict, strict=True)
if cfg.get("tie_word_embeddings", True):
model.lm_head.weight = model.embed_tokens.weight
model.eval()
prompt = "<|user|>\nExplain what a neural network is in simple terms.\n<|assistant|>\n"
print(model.generate(prompt, tokenizer, max_new_tokens=128))