Masinissa-73M

A 73.3M-parameter masked language model for Kabyle (Taqbaylit, kab, Latin script), trained from scratch on 3.92 billion tokens of deduplicated, provenance-tracked Kabyle text.

Masinissa-73M establishes the definitive state of the art on KabEncBench, the unified multitask benchmark for Kabyle representation learning. In a 2,000-bootstrap Bradley–Terry tournament across nine models anchored at FacebookAI/xlm-roberta-base = 1,000, Masinissa-73M achieves an Elo rating of 1,953.7 (Tier 1), leading the nearest external baseline by +753.4 Elo points.

Results: KabEncBench

All models are evaluated on KabEncBench (Track A: Frozen Scale-Normalized Probe) across six core linguistic tasks and three diagnostic probes. Backbones are 100% frozen; a layer-weighted scalar mix (ScalarMix) and linear probe are fitted across 5 random seeds per learning rate.

Bradley–Terry Elo Leaderboard

Pairwise Bradley–Terry Elo ratings are computed via 2,000 paired cluster-bootstraps over all task items, anchored at FacebookAI/xlm-roberta-base = 1,000:

Comprehensive Task Performance (%)

Part-of-Speech Tagging Detail (UD_Kabyle-ADPT)

On the gold Universal Dependencies UD_Kabyle-ADPT test split (845 sentences, 10,181 words, 8,563 whitespace tokens), Masinissa-73M scores 94.08% surface accuracy with a linear probe:

Stripping punctuation tokens isolates lexical and syntactic tagging performance:

Three things worth reading carefully:

The encoder is frozen. Only the probe is fitted. This measures what pretraining put into the representation, not what a deep task head can learn.

Both tokenisation settings are reported because they disagree. 29.5% of UD_Kabyle-ADPT words sit inside a multiword token, so "accuracy" is two different numbers depending on whether you score gold word segmentation or raw whitespace tokens. Reporting both makes this distinction transparent.

The corrupted condition tests orthographic resilience. Kabyle text in the wild carries systematic homoglyph substitution (Greek ε U+03B5 for Latin ɛ U+025B, γ for ɣ) in 2.6–3.2% of public corpora. Masinissa-73M retains 91.51% accuracy under corruption.

Intended use

Kabyle NLP where high-capacity contextual representations are required: sequence labelling, named entity recognition, dependency parsing, sentence similarity, semantic retrieval, and fine-tuning for classification tasks.

Not suitable for: generative text continuation (it is a bidirectional encoder), open-ended dialogue, translation, or non-Berber languages.

Usage

Loadable directly via transformers and torch with trust_remote_code=True:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

REPO = "agbalu/Masinissa-73M"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(REPO, trust_remote_code=True).eval()

text = "Axxam n [MASK] meqqer."
encoded = tokenizer(text, return_tensors="pt")
with torch.inference_mode():
    logits = model(**encoded).logits

position = (encoded["input_ids"][0] == tokenizer.mask_token_id).nonzero()[0, 0]
top = logits[0, position].topk(5).indices
print([tokenizer.decode(piece) for piece in top])

For extracting representations rather than logits:

from transformers import AutoModel

encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()
with torch.inference_mode():
    hidden = encoder(**tokenizer("Aman d tudert.", return_tensors="pt")).last_hidden_state
print(hidden.shape)  # (1, sequence_length, 512)

Normalise the input first. The vocabulary was built over text where ɛ is U+025B and ɣ is U+0263. Homoglyph-corrupted text fragments into fallback bytes and degrades downstream accuracy.

mammeri-tok-base-16k.model is the SentencePiece model the vocabulary was trained as, kept beside the converted tokenizer.json for anyone who needs the original.

Architecture

A modern deep bidirectional encoder featuring Block Attention Residuals and QK RMSNorm:

Training data

AƔBALU Corpus — deduplicated Kabyle documents tokenised to 16,000 pieces via Mammeri-Tok. Pretraining consumed 3,918,417,150 tokens (3.92B tokens) over 28,731 updates.

Every source carries an audited provenance record (source id, licence, retrieval date). Text passed the reference normaliser at version 1.3.0+rules1.0.0, which standardises orthography while preserving all phonemic and spirantised distinctions (ɣ ɛ ḥ ḍ ṣ ṭ ẓ ṛ č ǧ ţ).

Decontamination. Evaluated against FLORES+ kab_Latn and SIB-200 with zero overlap, verified against a positive-control detector.

Licence composition of the training text

The model weights and code are released under Apache-2.0. As with all web-derived language corpora, the weights licence does not relicense the underlying texts:

Training recipe

Pretraining loss

Validation cross-entropy reached 2.9303 nats (perplexity 18.73, accuracy 49.51% at step 27,231), improving monotonically from 6.81 at step 50.

Files

Reproduction

The source pretraining snapshot is step 27,231 (SHA-256 91083cdf72e04c8dc0bd9edff0334a9be5232f49bfdd92a2a354c1902ccb6b00), promoted under minimum validation cross-entropy (2.9303 nats, perplexity 18.73).

The downstream POS evaluation protocol trains a 20-epoch frozen linear probe across the 1,085 training sentences of UD_Kabyle-ADPT (seed 0) and evaluates on the 845 gold test sentences. All reported metrics reproduce deterministically under the fixed seed.

The name

Masinissa (c. 238 – c. 148 BCE) was the first king of Numidia, uniting the Massylii and Masaesyli into a unified North African state. Masinissa-73M represents the highest-capacity monolingual encoder trained for Kabyle to date.

Citation

@software{agbalu_masinissa_73m_2026,
  title  = {Masinissa-73M: a deep bidirectional masked language model for Kabyle},
  author = {AƔBALU},
  year   = {2026},
  url    = {https://huggingface.co/agbalu/Masinissa-73M},
  note   = {Trained on AƔBALU text corpus; normaliser 1.3.0+rules1.0.0}
}

Licence

Apache-2.0 for weights and code.