Emhotob-25M-v2

Emhotob is a family of small Arabic language models pretrained from scratch on Arabic web text. This is the 25M rung of the ladder (~25.27M parameters), part of a scaling series ranging from 500K to 25M parameters that all share the same tokenizer, context length, and training recipe. This is the v2 run: trained from scratch on ~5B tokens per model.

⚠️ These are tiny, research-scale models trained on a limited token budget. They are intended for scaling-law experiments, education, and Arabic NLP research — not for production use.

Model details

Property	Value
Architecture	Llama (decoder-only, RoPE, GQA)
Parameters	25,270,656 (~25.27M)
Hidden size	384
Layers	8
Attention heads	6 (KV heads: 3)
Intermediate size	1024
Context length	2048
Vocabulary	32,000 (custom Byte-Level BPE)
Tied embeddings	Yes
RoPE theta	10,000
Precision	bf16

Training

Property	Value
Data	`kaust-generative-ai/fineweb-edu-ar` (Arabic)
Tokens seen	~5B (1 epoch)
Optimizer	AdamW (fused), β=(0.9, 0.95), wd=0.1
LR schedule	6e-4, cosine, 2% warmup
Effective batch	128 sequences × 2048 tokens
Grad clipping	1.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "oddadmix/Emhotob-25M-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

prompt = "الذكاء الاصطناعي هو"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Given its size and limited pretraining budget, Emhotob-25M-v2 has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It has not been instruction-tuned or aligned, and no safety filtering has been applied. Use accordingly.