Nawah-RuleCheck-BERT-6M-v2

Arabic rule checker on a bidirectional BERT encoder: give it a text and a rule in ordinary Arabic, it answers مطابق or مخالف. 5,961,602 parameters, fine-tuned from Nawah-BERT-6M-v2 (15B pretraining tokens) with the RuleCheck project's own script and scored by its own eval_all.py.

Results

The published ladder is Llama decoders; this is the encoder arm. The 5M rung was re-run locally and reproduced its card exactly, which validates the harness.

model	base	params	seen wording	unseen wording	min-pairs	hand-written
RuleCheck-500K	Llama	518K	0.9866	0.9778	0.9926	0.5558
RuleCheck-1M	Llama	1.07M	0.9952	0.9864	0.9991	0.7708
RuleCheck-5M	Llama	5.08M	0.9979	0.9949	0.9926	0.8642
RuleCheck-v2	Llama	51.8M	0.9980	0.9980	0.9991	0.8883
BERT-6M v1 (5B pretrain)	BERT	5.96M	0.9534	0.9357	0.9898	0.9117
this model, v2 (15B pretrain)	BERT	5.96M	0.9600	0.9598	0.9907	0.8900
majority baseline	—	—	0.6401	0.6401	0.5000	0.5925

Better pretraining bought in-distribution accuracy and cost out-of-distribution robustness. Against v1: unseen wording +2.4 points (0.9357 → 0.9598), seen +0.7, minimal pairs +0.1 — but hand-written wording −2.2 (0.9117 → 0.8900).

Both BERT rungs still beat every Llama rung on hand-written wording, the column the original card calls the honest hard one — v1 by 2.3 points over the 51.8M model, v2 by 0.2. That the weaker encoder generalises better to unseen phrasing is counter-intuitive and rests on one run each; treat it as a signal, not a finding.

Trained with the project's own hyperparameters (lr 3e-4, 3 epochs, batch 64), tuned for a decoder, so this is likely under-tuned.

Usage — mean pooling is required

The base is pretrained on packed 2,048-token chunks with no [CLS]; BertForSequenceClassification's position-0 pooler is meaningless here and randomly initialised, and the model then never leaves the class prior at any learning rate from 1e-4 to 3e-3. Load with the shipped bert_meanpool.py:

import torch
from transformers import AutoTokenizer
from bert_meanpool import MeanPoolBertForSequenceClassification

M = "oddadmix/Nawah-RuleCheck-BERT-6M-v2"
tok = AutoTokenizer.from_pretrained(M)
model = MeanPoolBertForSequenceClassification.from_pretrained(M, dtype=torch.float32).eval()

text = "أهلاً، هل تتوفر خدمة التوصيل السريع داخل الدوحة اليوم؟"
rule = "يُمنع ظهور الحروف اللاتينية داخل النص"
enc = tok(f"النص: {text}\nالقاعدة: {rule}", return_tensors="pt",
          add_special_tokens=False, truncation=True, max_length=224)
with torch.no_grad():
    print(["مخالف", "مطابق"][model(**enc).logits.argmax(-1).item()])

The template and add_special_tokens=False must match training exactly.

Ships the full harness: bert_meanpool.py, train_rules_cls.py, eval_all.py, rules_common.py, the frozen cf_cases.json / ood_cases.json, and eval_results.json with the per-rule breakdown.