Emhotob-500K-v2
Emhotob is a family of small Arabic language models pretrained from scratch on Arabic web text. This is the 500K rung of the ladder (~518K parameters), part of a scaling series ranging from 500K to 25M parameters that all share the same tokenizer, context length, and training recipe. This is the v2 run: trained from scratch on ~5B tokens per model.
⚠️ These are tiny, research-scale models trained on a limited token budget. They are intended for scaling-law experiments, education, and Arabic NLP research — not for production use.
Model details
| Property | Value |
|---|---|
| Architecture | Llama (decoder-only, RoPE, GQA) |
| Parameters | 518,224 (~518K) |
| Hidden size | 16 |
| Layers | 2 |
| Attention heads | 2 (KV heads: 1) |
| Intermediate size | 48 |
| Context length | 2048 |
| Vocabulary | 32,000 (custom Byte-Level BPE) |
| Tied embeddings | Yes |
| RoPE theta | 10,000 |
| Precision | bf16 |
Training
| Property | Value |
|---|---|
| Data | kaust-generative-ai/fineweb-edu-ar (Arabic) |
| Tokens seen | ~5B (1 epoch) |
| Optimizer | AdamW (fused), β=(0.9, 0.95), wd=0.1 |
| LR schedule | 6e-4, cosine, 2% warmup |
| Effective batch | 128 sequences × 2048 tokens |
| Grad clipping | 1.0 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "oddadmix/Emhotob-500K-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
prompt = "الذكاء الاصطناعي هو"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
Given its size and limited pretraining budget, Emhotob-500K-v2 has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It has not been instruction-tuned or aligned, and no safety filtering has been applied. Use accordingly.
© SupraLabs 2026 — PROJECT EMHOTOB.