NADI 2026 ADI-20 — Whisper-small encoder

The strongest single system in this release: the Whisper-small encoder with mean pooling and a 20-way linear head, fully fine-tuned on ADI-20.

Part of the NAMAA at NADI 2026 Shared Task collection.

At a glance

Results

All four systems in this release share one training recipe and one evaluation protocol, so these numbers are directly comparable.

Two results worth noting. Parameter count did not predict quality — XLS-R 300M is 3.6x the size of the Whisper encoder and lands 18.8 points behind it; pretraining domain mattered more than capacity. And combination paid more than any single architecture choice: the four-way weighted soft vote reported in the team paper reaches 95.00% development accuracy against 85.53% for the best single member, by adding two public classifiers to the two released here.

_{The two-way row was measured on 2026-07-18 against an earlier Whisper checkpoint (dev 81.07%), where it beat its better member by 4.51 points. The Whisper weights released here are the later step-90,000 checkpoint at 85.53%, so that blend has not been re-measured against them. The four-way row is the paper's development figure; its members and weights are described in nadi26-adi20-ensemble. Neither is a blind-set number — see below.}

Official NADI 2026 result

The numbers above are development accuracy. The shared task was scored on a held-out blind set, and that gap is the headline finding of this release:

Organisers describe the blind evaluation as out of domain: it need not cover every class, and the recordings differ from the ADI-20 micro training material. A system at 95% on validation landing at 37.7% on the blind set is the honest measure of how far ADI-20-micro-trained dialect ID generalises — and the reason the per-model development numbers on this page should be read as in-domain results, not as a capability claim.

For reference, the organisers' own development baseline scores 81.19%, so the single systems here are competitive in-domain; the collapse is a shift problem, not a fitting problem.

_{Official blind figures from the team paper, CodaBench submission 861912. Blind-set feedback influenced which of ten submissions was retained, so the selected score is not an independent estimate of a pre-specified rule.}

Usage

import torch, librosa
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification

REPO = "NAMAA-Space/nadi26-adi20-whisper-small"
fe    = AutoFeatureExtractor.from_pretrained(REPO)
model = AutoModelForAudioClassification.from_pretrained(REPO).eval()

wav, _ = librosa.load("clip.wav", sr=16_000, mono=True)
wav = wav[:16_000 * 30]                      # scored on the first 30 s, as in training

inputs = fe(wav, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]

top = probs.argsort(descending=True)[:3]
for i in top:
    print("{:<4} {:.3f}".format(model.config.id2label[int(i)], float(probs[i])))

Batched inference works the same way — pass a list of waveforms to the feature extractor — Whisper pads every clip to the same 30 s window.

Label set

Class order is fixed; index i in the output vector is the code below.

How it was trained

A single multi-backbone recipe trained all four systems in this release, so the numbers above compare architectures rather than training tricks. Every run used the same data, schedule length, augmentation, metric and seed.

Evaluation protocol

Both metrics come from the validation split of UBC-NLP/NADI_2026_ADI20_micro (10,806 clips), scored with the NADI baseline's own evaluation code:

• Accuracy — top-1 over the 20 classes.
• Average cost — the NIST LRE-style detection cost: for each target class, the log-likelihood ratio against all others is thresholded at the Bayes point for p_target = 0.5, C_miss = C_fa = 1, and the per-class false-alarm and miss rates are averaged. Lower is better; it penalises confident mistakes that accuracy alone hides.

Audio is capped at 30 s at evaluation time (the first 30 s of any longer clip is used). No augmentation is applied at evaluation.

These are validation numbers, not blind-test numbers. The NADI 2026 blind test labels are held by the organisers; nothing in this repo was tuned on them.

What changed relative to the raw training checkpoint

The original artefact was a best.pt produced by torch.save mid-training: it carried the optimizer moments, the LR-scheduler state and the Python/NumPy/CUDA RNG state alongside the weights, and it only loaded into a bespoke backbone + pooling + Linear wrapper class.

This repo contains the weights only, in safetensors, remapped onto the stock WhisperForAudioClassification class. Two things follow: the download is roughly half the size, and the model loads with from_pretrained without any custom code or pickle execution.

The remap is numerically exact, not approximate. The stock head computes classifier(mean(projector(h))) while the trained module computed head(mean(h)). Projection and mean-pooling are both linear, so they commute: setting classifier_proj_size = 768 and freezing projector to the identity makes the two computation graphs the same function. This was verified by loading the same trained weights into both and comparing logits — the maximum absolute difference was exactly 0.0 -- the two computation graphs are bit-identical.

The identity projector is the only tensor in this repo that was not trained: it adds 590,592 parameters that compute a no-op, so the 88,169,492 trained parameters are unchanged and model.safetensors holds 88,760,084 in total.

If you need the optimizer state to resume training, it is in the original private checkpoint; ask the authors.

A conflict between this checkpoint and the paper

Section 3.3 and Appendix B.1 of the paper describe this system as a Whisper-large-v3 classifier adapted with rank-32 LoRA (alpha 64, dropout 0.05, adapters in both encoder and decoder). The tensors in this repository are a fully fine-tuned openai/whisper-small encoder: 12 encoder layers, d_model 768, 80 mel bins, 1,500 positional embeddings, no decoder, and no LoRA or adapter tensors of any kind.

The checkpoint is nonetheless the one the paper reports. Its stored best record reads acc = 85.5266, which is the paper's 85.53%, and the paper names FatimahEmadEldin/nadi26-whisper-ckpts as the source. The paper's own limitations note that "pooling/head details for our Whisper classifier" could not be recovered from the records, so the architecture prose appears to be a reconstruction error rather than a different model.

The weights are authoritative. What you download here is whisper-small, fully fine-tuned.

This section exists because a model card that silently disagreed with its own paper would waste someone's afternoon.

Intended use and limitations

Intended use. Research on spoken Arabic dialect identification, NADI 2026 Subtask 2 reproduction, and as a component or baseline in Arabic speech pipelines.

Known limitations.

• Country is not dialect. The labels are the speaker's country of origin as annotated in ADI-20. Dialect boundaries cross national borders and several Levantine and Gulf varieties are mutually near-identical in short clips; the confusions the model makes often reflect genuine linguistic overlap rather than model failure.
• Domain. ADI-20 is built from broadcast and online speech. Accuracy on telephone audio, far-field recordings, heavy code-switching or sung speech is untested.
• Clip length. Evaluation caps audio at 30 s and training cropped to short windows. Very short clips (under ~3 s) carry little dialectal evidence and accuracy degrades.
• Class balance. Per-class performance is not uniform; MSA and the high-resource dialects are easier than the low-resource ones. Do not read the aggregate accuracy as a per-country guarantee.
• Not a speaker or language detector. Non-Arabic input will still be forced into one of the 20 Arabic classes.

Out of scope. Do not use this to infer a person's nationality, residency, ethnicity or immigration status. Accent is not identity, the label space is a coarse proxy, and errors fall hardest on the varieties that are already least represented.

Provenance and citation

This model is part of the NAMAA Community submission to the NADI 2026 shared task, by Mohamed Abdelaziz, Fatima Emad Eldin, Mousa Ashraf Abd El Malak and Omer Nacar, released through NAMAA-Space (Network for Advancing Modern ArabicNLP & AI). Cite the team paper, not this repository alone.

@inproceedings{namaa_nadi2026,
  title     = {NAMAA Community at NADI 2026 shared task: Mixed-Dialect and
               Code-Switched Arabic ASR and Dialect Identification},
  author    = {Abdelaziz, Mohamed and Emad Eldin, Fatima and
               Abd El Malak, Mousa Ashraf and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing
               Conference (ArabicNLP 2026)},
  year      = {2026}
}

Please also cite the NADI 2026 shared-task overview (Sullivan et al., 2026), the ADI-20 dataset (Elleuch et al., 2025), and the base model openai/whisper-small.