NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments

KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026

This is the development-phase experiment record of team NAMAA. It holds the training and inference notebooks, the per-run metrics and configurations, the Konooz development-set predictions and submissions for every candidate system, the evaluation scripts, and the software environment.

How this relates to the paper. These experiments come before the submitted systems described in the camera-ready paper. The submitted AdaptNER system ranked 1st (test micro-F1 75.96). It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and ARBERTv2 + GlobalPointer. The submitted RE system ranked 2nd (93.48). It is a majority vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables are in §2.

The fine-tuned weights are in their own model repositories:

All scores are entity-level strict micro-F1.

---

Repository layout

├── experiments/                      per-run metrics + full config (repro_bundle.json)
│   ├── EXP-001_AraBERTv02_Flat/
│   ├── EXP-001_AraBERTv02_Nested/
│   ├── EXP-002_MARBERTv2_Flat/       weights not released; test predictions kept here
│   └── EXP-002_MARBERTv2_Nested/
├── notebooks/
│   ├── training/                     Wojood fine-tuning, one notebook per run
│   ├── konooz_dev_inference/         load weights from the Hub, predict Konooz dev, write submission
│   ├── blind_test_inference/         blind-test inference for the ARBERTv2 (unweighted) system
│   └── eda/                          Wojood + Konooz exploratory analysis
├── konooz_dev_predictions/
│   ├── namaa_wojood_runs/            EXP-001/002 predictions + submission zips
│   ├── backbone_sweep/               the nested backbone comparison (paper Table 4)
│   └── ensembles_and_llm/            top-3 / top-4 voting ensembles, LLM run
├── evaluation/                       per-entity-type and per-domain scorers + results
├── scripts/                          ensemble voting, Konooz EDA, hosted-LLM baseline
└── environment/pip_freeze.txt

---

1. Wojood experiments (in-domain)

Same recipe for all four runs, taken from repro_bundle.json: 8 epochs, learning rate 3e-5, batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold. The flat runs use a softmax BIO head.

The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on different variants of Wojood, so compare them as indicative only.

Per-entity-type macro-F1 on the Wojood test split (evaluation/pertype_results.json):

MARBERTv2 nested makes no predictions at all for PERCENT, PRODUCT, UNIT and QUANTITY, so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13 Wojood sub-corpora. evaluation/compute_pertype.py and compute_perdomain.py regenerate these numbers from each model's test_predictions_conll.txt. The recomputed precision matches the original scorer exactly, and micro-F1 matches it to within 0.001.

---

2. Official test results (camera-ready paper)

Entity-level micro-averages under exact span-and-type matching, in percent.

Subtask 1 — AdaptNER (Konooz test). Ranked 1st.

Subtask 2 — Relation Extraction (WojoodRelations). Ranked 2nd.

The checkpoints for these submitted systems are not in this repository.

3. Development-phase Konooz leaderboard

These are the team's scores on the Konooz development set (100 sentences) during development. They are not reported in the camera-ready paper.

† Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74).

backbone_sweep/ also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full). These two runs were scored only on Wojood validation (paper Table 4) and were not submitted to the Konooz leaderboard.

Submission format: one token per line, followed by 21 space-separated IOB2 columns in this order:

CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP
OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE

---

4. Reproducing

1. Get Wojood 1.1 and the Konooz development and blind-test data from SinaLab, Birzeit University. They are not redistributed here.
2. Training: run notebooks/training/<run>.ipynb. They were written for Google Colab (A100 or T4). Each run's exact configuration and library versions are in experiments/<run>/repro_bundle.json.
3. Konooz inference: notebooks/konooz_dev_inference/ loads the released checkpoints from the Hub by name.
4. Ensemble: scripts/ensemble_predictions.py does a weighted vote over per-system prediction files.

The notebooks read the Hub token from a Colab Secret or the HF_TOKEN environment variable. No credentials are stored in this repository.

---

Citation

@inproceedings{namaa-knowledgegrapheval-2026,
  title     = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained
               {LLM} Ensembles for {A}rabic Knowledge Graph Construction},
  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and
               A.azim, Mona and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked above.

License

Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those terms if you redistribute them.