ArabicBertHash v2

ArabicBertHash v2 is an Arabic masked-language encoder with 49.4M parameters. It combines compact learned token representations with a six-layer transformer for masked-word prediction and text-feature extraction.

Each token uses an 8-dimensional learned code projected into a 512-dimensional hidden representation. This reduces the input embedding module to 516K parameters, while retaining a full-vocabulary MLM output head.

Quick start

Tested with Python 3.13, PyTorch 2.13.0 and Transformers 5.14.1. Install the versions in requirements.txt with a PyTorch build suitable for your CPU or GPU. Authenticate with hf auth login to access the private repository.

The model uses custom Transformers code, so loading requires trust_remote_code=True. Review the included code and pin a repository revision for reproducible use.

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "Abdelkareem/arabic-berthash-v2"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()

text = "اللغة العربية هي لغة [MASK] الكريم"
batch = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024)
mask = batch["input_ids"].eq(tokenizer.mask_token_id)
assert mask.sum().item() == 1

with torch.inference_mode():
    logits = model(**batch).logits[mask][0]
    probabilities, token_ids = logits.softmax(-1).topk(5)

for token_id, probability in zip(token_ids.tolist(), probabilities.tolist()):
    filled = batch["input_ids"][0].clone()
    filled[mask[0]] = token_id
    print(tokenizer.decode(filled, skip_special_tokens=True), probability)

Decode the completed sentence to inspect Arabic suffixes and spacing. Softmax probabilities describe token predictions; they are not calibrated factual-confidence scores. For GPU inference, move both the model and input tensors to the same device. A standalone example is available in example.py.

Extract text features

The encoder returns a 512-dimensional vector per token. This example produces a normalized sentence vector by mean-pooling non-special tokens:

import torch.nn.functional as F

batch = tokenizer(
    ["هذا مثال عربي.", "نص آخر للمقارنة."],
    padding=True, truncation=True, max_length=1024, return_tensors="pt",
)
with torch.inference_mode():
    hidden = model.bert(**batch).last_hidden_state
    valid = batch["attention_mask"].bool()
    special_ids = torch.tensor(tokenizer.all_special_ids, device=hidden.device)
    valid &= ~torch.isin(batch["input_ids"], special_ids)
    pooled = (hidden * valid.unsqueeze(-1)).sum(1)
    pooled /= valid.sum(1).clamp_min(1).unsqueeze(-1)
    vectors = F.normalize(pooled.float(), dim=-1)

For classification or retrieval, train and validate a task head or projection on your data. This checkpoint provides an MLM head; the pooling example does not include the learned projection used in the retrieval evaluation below.

Architecture

Each vocabulary token has its own learned code. The shared projection expands that code into the encoder's hidden dimension.

Parameter breakdown

The input embedding module is approximately 63.5× smaller than a dense 64,001×512 embedding table. This is a component-level reduction: the untied output head accounts for most of the complete model's parameters.

Pretraining used 1,024-token windows. The configuration allows 2,048 positions, but longer-context quality has not been evaluated.

Tokenizer

The included tokenizer has a 64K base vocabulary plus [MASK]. Use it directly on ordinary Arabic text; no external Arabic segmentation step is required.

The serialized tokenizer defines normalization and tokenization. Keep it paired with these weights when training or evaluating the model.

Training

Data

The training corpus combined Arabic Wikipedia and FineWeb-2, prepared as 51,492,343 tokenized examples. The recorded source-building configuration references:

• HuggingFaceFW/fineweb-2, Arabic-script subsets arb_Arab, ary_Arab, arz_Arab, apc_Arab and acm_Arab.
• wikimedia/wikipedia, Arabic 20231101.ar.

Exact source revisions, final mixture proportions and unique-document counts are not fully recorded. Training/evaluation overlap has not been exhaustively ruled out.

Recipe

Training covered approximately 90.88M example passes. The batch/context calculation gives 93.06B nominal token positions, including repetition and padding where applicable; this is not a count of unique or attended content tokens.

Saved trainer-state fields and the training recipe are available in training_summary.json.

Evaluation

Luna and Terra comparison

Scores show Luna → Terra percentages on the same 100 Arabic examples per task. NLI and retrieval use the same frozen probe protocol for every checkpoint. Context values are configuration limits. The ArabicBertHash v2 release tokenizer is set to 1,024 tokens.

How the tasks were measured

• MLM: one shared-tokenizer-eligible word is masked per question. Top-1 accepts the exact reference or a completion judged contextually acceptable; top-5 accepts at least one such completion. Uncertain judgments are excluded. Candidates are reviewed in their completed sentences with model identity and rank hidden. For this model, judge-acceptable-only scores were 62%/87%; adding exact-reference matches gives the reported 65%/93%.
• Classification: three-way natural language inference, using frozen CLS features and a linear probe trained for 400 updates on 4,096 pairs, with 512 development and 1,024 test pairs. The reviewed score measures agreement with each judge’s labels on 100 cases. It is not general classification accuracy.
• Retrieval: frozen mean-pooled features with a projection trained for 300 updates on filtered NLI-derived triplets. Evaluation uses 512 development queries, 512 test queries and a 1,024-sentence gallery; Each judge reviews the retrieved top-five passages for 100 test queries. The released MLM weights do not include this evaluation projection.

Exact versus reviewed scores

On the paired 100-case subsets:

Reviewed MLM includes plausible alternatives, while reviewed NLI and retrieval use the judge's labels/relevance judgments. These columns therefore measure different targets. In NLI, one judge label was uncertain and 24 confident labels disagreed with the dataset labels; automated review is not adjudicated ground truth.

On the full exact-reference diagnostic sets:

Full-set scores and 100-case subset scores have different denominators. Machine-readable results are available in evaluation/summary.json, with protocol details in evaluation/protocol.md.

Evaluation limitations

These are small, previously exposed research diagnostics with automated judgments. The judge was checked on small synthetic Arabic fixtures, not independently validated by human annotators. Most training comparisons use one seed, so small differences should not be treated as reliable general rankings.

Models share evaluation examples and probe-training budgets, but differ in pretraining data, compute and task exposure. Some comparison models already received NLI/retrieval training. Retrieval uses NLI-derived sentences rather than a broad Arabic passage-search benchmark. MLM acceptance measures contextual plausibility and may allow a different meaning from the reference.

Intended use and limitations

The model supports research on compact Arabic encoders, masked-word completion and task-specific adaptation.

• Train and evaluate task heads for classification or retrieval before deployment.
• Test on representative Arabic domains and dialects; their coverage is uneven and not fully quantified.
• Independently verify factual and numerical outputs.
• Web/Wikipedia training data can contain errors, personal information, stereotypes and offensive material. Dedicated privacy, toxicity and fairness evaluations have not been completed.
• Speed, energy use and deployment efficiency have not been benchmarked. The smaller input embedding module does not establish whole-model efficiency over MiniBERT or other encoders.

Files and reproducibility

Pin the repository revision when reporting experiments. The included evaluation artifacts contain scores, hashes and case identifiers; raw evaluation texts and complete probe-training data are not distributed here.

License and citation

Code and model weights are licensed under Apache-2.0. See LICENSE and NOTICE. Source texts and third-party components retain their respective licenses.

Developed by Karem Elkhateb, Hugging Face account Abdelkareem.

@misc{arabicberthashv2,
  author = {Elkhateb, Karem},
  title = {ArabicBertHash v2},
  year = {2026},
  howpublished = {Hugging Face model repository},
  url = {https://huggingface.co/Abdelkareem/arabic-berthash-v2},
  note = {Checkpoint 1420000; cite the repository revision used}
}