Icelandic ModernBERT, size base

Paper: Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters (arXiv:2610.36230). If you use this model, please cite the paper. The BibTeX entry is at the end of this card.

Question this run answers. How good does a ModernBERT-architecture encoder trained only on Icelandic get at this size when it sees 4x the Chinchilla-optimal number of tokens (80 tokens per parameter)? It is one rung of a size ladder (xs, s, m, base, large) trained with one recipe, so the rungs can be compared with each other and with existing Icelandic encoders (IceBERT, ELECTRA/ConvBERT-IGC, XLM-R, mmBERT, ScandiBERT) under one fine-tuning protocol.

Release. Released under the MIT licence with the paper, as one rung of its scratch-trained size ladder (xs to large, at 80 and 160 tokens per parameter); the training corpus is not released, because not every source states a licence that permits redistribution. Parts of the training data carry licences that require attribution (CC BY 4.0: IGC; CC BY-SA: Wikipedia, Wikisource) or come from web crawls with mixed rights (FineWeb-2, FinePDFs, HPLT 3.0, MaCoCu, MADLAD-400; IC3-v2 states no licence). The IGC restricted-licence parts (News2, Books) were not used. MIM-GOLD-NER, used only for evaluation, has a research-only licence, so no model fine-tuned on it is released.

Model

Fill-mask. Write the mask token right after the previous word, with no space: Reykjavík er höfuðborg[MASK]. The tokenizer keeps each word's leading space inside the word's token, so a space typed before the mask becomes a token of its own. In training such a lone space comes mostly before a number, because digits are split off, so with the space the model predicts digits.

Training

Data

Deduplicated (exact + MinHash 0.75 across all sources, curated sources kept first), quality- and language-filtered, and decontaminated against every evaluation set (13-gram and whole-sentence matching). Per-source epochs below are for the 1,024-token phase.

Sources: IGC-2024 open part (arnastofnun/IGC-2024, CC BY 4.0); the non-IGC subsets of danish-foundation-models/icelandic-dynaword (government gazette and Rafbókavefur under Icelandic copyright law's exemptions; Wikipedia, Wikisource, Wikibooks, wiki comments, radio talks CC BY-SA; sagas CC BY; Gutenberg); Icelandic Wikipedia (wikimedia/wikipedia 20231101.is); MaCoCu-is 2.0 (CC0); FineWeb-2 and FinePDFs isl_Latn (ODC-By); HPLT 3.0 isl_Latn; MADLAD-400 is, clean split (ODC-By); IC3-v2 (Miðeind; licence not stated, Common Crawl derived). IcePaHC excluded (evaluation source); restricted IGC parts (News2, Books) not used.

Results

Held-out MLM (15% masking) loss per source at the end of training: {"dynaword": 0.626, "finepdfs": 1.467, "fineweb2": 1.346, "hplt3": 1.382, "ic3v2": 0.972, "igc": 0.961, "macocu": 1.252, "madlad": 1.124, "wiki": 0.884}.

Fine-tuning (same protocol for every model: learning rate picked on dev from a 3-value grid with seed 0, then 5 seeds at that rate; best-dev epoch; 95% CI from a hierarchical bootstrap over seeds and test items):

Tasks: NER = MIM-GOLD-NER (IceBERT split, entity micro F1); POS = UD Icelandic-Modern + GC (UPOS accuracy); ScaLA = acceptability built from the same UD splits (MCC); IceEC = error vs no error (MCC); QA = NQiI + RUQuAD-1 (token F1).

Caveats

• One pretraining seed per size; differences between sizes include pretraining-seed noise that the CIs do not cover.
• The 4x Chinchilla budget is a decoder-derived heuristic; no compute-optimal law for MLM encoders is established.
• Larger sizes repeat the data several times (see epochs); returns from repetition fall off beyond about 4 epochs (Muennighoff et al. 2023).
• IGC JSONL documents were not used for the 8,192-token phase because their paragraph order is not guaranteed.
• AdamW with gradient clipping replaces ModernBERT's StableAdamW; global RoPE theta is 160k from the start.
• The fine-tuning learning rate comes from a 3-value grid per size class. Where the best value sits at the top of the grid (e.g. large: NER, POS, QA at 4e-5), a wider grid might raise those scores a little.
• Fine-tuning used bf16 autocast; ScandiBERT gave non-finite losses in bf16 and was fine-tuned in fp32 (flagged per run).

Full numbers: the paper, arXiv:2610.36230. The fine-tuning results above are the first round; the paper extends every learning-rate grid until the chosen rate is not at an edge, so its numbers differ slightly. Export parity check (ours vs transformers logits): see export_check.json.

Citation

@misc{einarsson2026icebertopendatamodernbertencoders,
  title         = {Beyond {IceBERT}: Open-Data {ModernBERT} Encoders for {Icelandic} from {12M} to {1B} Parameters},
  author        = {Hafsteinn Einarsson},
  year          = {2026},
  eprint        = {2610.36230},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.36230}
}