Nawah-Router-BERT-6M-bilingual — a router head trained on English and Arabic together

Give it a text and any categories in plain English or Arabic; it scores all of them in one forward pass. No fixed taxonomy, no separate model per language. 5,977,985 parameters.

The headline finding: English-only training did not work on this backbone

Nawah-BERT-6M-v2 was pretrained exclusively on Arabic MSA text (fineweb-edu-ar). Training a routing head on prompt-routing-en alone, same code, same hyperparameters, produced a model that learned nothing: training loss sat flat at ~1.57 for all 3 epochs, and every eval split landed at its random baseline.

Mixing in the Arabic corpus rescued it — not just held Arabic steady, but pulled English routing from pure chance to well above it:

eval split	English-only	this model — English	this model — Arabic	Arabic-only (v2)
unseen_lanes	0.210 (=random)	0.859	0.929	0.933
unseen_domain	0.239 (=random)	0.590	0.694	0.701
unseen_axis	0.252 (=random)	0.547	0.605	0.600
hard	0.213 (=random)	0.839	0.903	0.911

All four numbers were re-measured in the same session with eval_router_bilingual.py; the Arabic columns reproduce the published Nawah-Router-BERT-6M-v2 card within normal run-to-run variance, which is what makes the English-only comparison trustworthy.

Why joint training fixes it is not fully confirmed. The working hypothesis: the shared 2-layer scoring head, trained only on English gradients, got stuck in a degenerate optimum that predicts a near-uniform distribution over categories regardless of content (consistent with the flat loss staying near ln(mean lane count) ≈ 1.6 for all three epochs). The Arabic gradients, on representations the backbone was actually pretrained to produce, appear to pull the head out of that optimum early enough that it then also learns from the weaker — but not degenerate — English signal. This is a hypothesis, not a verified mechanism; it has not been tested against, say, a lower learning rate or longer warmup on English alone.

Practical takeaway

If you need English routing on a tiny Arabic-pretrained backbone, training on English data alone may not be enough — pairing it with a related-task corpus in the backbone's pretraining language was what worked here, on this backbone, at this scale. Don't assume that generalizes without testing on your own data.

Usage

from transformers import AutoTokenizer
from routing_model_bilingual import RouterModel, route      # ships in this repo

M = "oddadmix/Nawah-Router-BERT-6M-bilingual"
tok = AutoTokenizer.from_pretrained(M)
model = RouterModel.from_pretrained(M)

route(model, tok, "The order is an hour late and the driver isn't answering",
      ["delivery enquiry", "delayed-order complaint", "payment issue"])
# lang="en"/"ar"/"auto" (default) - auto picks by script

Trained on oddadmix/prompt-routing-en (173,223 English rows) + oddadmix/arabic-prompt-routing (207,097 Arabic rows) mixed in one training run, 3 epochs, with train_router_head_multi.py's defaults (LR 3e-4, batch 32). Each row is wrapped in its own language's template ("Text:/Categories:" or "النص:/الفئات:") before the shared span-pooling head scores it — see that script's docstring.