MORENA 1.5B base
The pretrained and mid-trained base model of the MORENA family: a 1.5B-parameter decoder trained from scratch on 251.7B tokens for twelve Latin-script African languages plus English, French and code, then annealed for 63B tokens on higher-quality and task data. This is not a chat model. It is the recommended starting point for fine-tuning, continued pretraining and domain adaptation in these languages.
Headline numbers
| MORENA 1.5B base | reference | |
|---|---|---|
| African bits per byte, mean of 12 (lower is better) | 1.408, lowest of 26 models measured | Lugha-Llama-8B 1.423, gemma-3-12b-it 2.159 |
| Translation, FLORES+ chrF++, English into 5 African languages, 3-shot | 43.2 | MADLAD-400-3B 37.8, Lugha-Llama-8B 36.8 |
| Translation, 5 African languages into English, 3-shot | 47.8 | MADLAD-400-3B 53.6, Lugha-Llama-8B 52.4 |
| ARC-easy / ARC-challenge (English, zero-shot cloze, 250 items each) | 0.676 / 0.412 | Qwen2.5-1.5B 0.668 / 0.440, gemma-3-1b 0.696 / 0.344 |
| MMLU | 0.284 | Qwen2.5-1.5B 0.324; close to the 0.25 floor, factual knowledge was traded away deliberately |
The advantage over gemma-3-1b-pt is not a translationese artefact: it is the same on native news (1.56x) and speech transcripts (1.55x) as on FLORES (1.61x). No safety evaluation applies to this checkpoint; it has not been instruction-tuned.
Files
model.safetensors (bf16), config.json, tokenizer.json, modeling_morena.py, load_example.py,
SHA256SUMS. Architecture: 28 layers x 2048, GQA 16/4, SwiGLU 6144, RoPE theta 500,000, 4,096
context, tied embeddings, 65,536-entry tokenizer trained on the target mix.
Training
251.7B tokens of pretraining across 60,000 steps, then 63B tokens of mid-training: 315B tokens seen in total, 12,661 A100 GPU-hours (about 20% useful utilisation across the whole run, 28 to 33% in steady state during pretraining). Training mixture moved in three regimes as machine-translated languages landed: African text was 24.8% of tokens seen (14.8% machine-translated) for steps 1 to 25,304, 39.1% (31.0% machine-translated) for steps 25,305 to 60,000, and 50.2% (41.6% machine-translated) during mid-training. Nine languages were machine-translated from English documents: this is 24% of pretraining tokens seen and 28% including mid-training (57.1B tokens on disk, 21% of the 271B on disk).
Optimiser: Muon for non-embedding weights, AdamW for the rest, warmup-stable-decay schedule.
The MORENA family
| model | params | African bpb (all 12, lower is better) | role |
|---|---|---|---|
| MORENA 1.5B base | 1.485B | 1.408 | pretrained and mid-trained; fine-tuning starting point |
| MORENA 1.5B instruct | 1.485B | 1.441 | chat, translation, tool calling; the model described in the paper |
| MORENA 0.5B mini | 503M | 1.520 | pruned and distilled from the 1.5B base |
| MORENA 0.5B mini instruct | 503M | 1.540 | chat fine-tune of the mini |
| MORENA 0.2B nano | 209M | 1.583 | cheap trunk for ASR rescoring, keyboards, normalisation |
26 models in total were measured on African bits per byte, 21 of them outside models from 125M to 12B parameters. MORENA 1.5B base is lowest of all 26 at 1.408; the 8B African specialist Lugha-Llama-8B is second at 1.423, ahead of the other four MORENA sizes on the twelve-language mean. Every general-purpose model measured, from 1B to 12B parameters, trails well behind: the best of them, gemma-3-12b-it, reaches only 2.159. Twelve languages: Shona, Swahili, Hausa, Yoruba, Igbo, isiZulu, isiXhosa, Kinyarwanda, Setswana, Afrikaans, isiNdebele and Nigerian Pidgin, plus English, French and code. isiNdebele (ISO code nbl) has no FLORES+ or MasakhaNEWS coverage and is evaluated on NCHLT transcripts only.
Tokenizer: 65,536-entry byte-fallback BPE trained on the target mix. African text costs 0.249 tokens per byte against 0.234 for English, about 6% more per byte than English in MORENA's vocabulary, but that same African text needs 1.39x fewer tokens than under Gemma 3's vocabulary and 1.53x fewer than under Llama 3.2's.
Translation is measured for seventeen models in total, thirteen of them outside models.
Author and citation
Isheanesu Misi, Vambo AI. Trained on CINECA Leonardo, with support from the AI Hub for Sustainable Development.
@techreport{misi2026morena,
title = {MORENA: An African Foundation Model},
author = {Misi, Isheanesu},
institution = {Vambo AI},
year = {2026},
month = {September},
note = {Released 18 September 2026}
}
Licence
Apache 2.0. Commercial use is permitted, including fine-tuning and redistributing derivatives, under the terms of the licence.
The training corpus is not released with these weights.