Icelandic ModernBERT, size base-160tpp

Paper: Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters (arXiv:2610.36230). If you use this model, please cite the paper. The BibTeX entry is at the end of this card.

Question this run answers. What does doubling the training tokens buy? This is the 160 tokens/parameter arm (8x Chinchilla) of the base model. It has the same architecture, tokenizer, schedule shape (3% warmup, 15% 1-sqrt decay, last 10% at 8,192 tokens) and seed as hafsteinn/icelandic-modernbert-base (80 tokens/parameter). The only other change is the CulturaX unique remainder (+0.34B tokens, +3.7%) added to the mix to reduce repetition. It is one rung of a size ladder (xs, s, m, base, large) trained with one recipe, so the rungs can be compared with each other and with existing Icelandic encoders (IceBERT, ELECTRA/ConvBERT-IGC, XLM-R, mmBERT, ScandiBERT) under one fine-tuning protocol.

Release. Released under the MIT licence with the paper, as one rung of its scratch-trained size ladder (xs to large, at 80 and 160 tokens per parameter); the training corpus is not released, because not every source states a licence that permits redistribution. Parts of the training data carry licences that require attribution (CC BY 4.0: IGC; CC BY-SA: Wikipedia, Wikisource) or come from web crawls with mixed rights (FineWeb-2, FinePDFs, HPLT 3.0, MaCoCu, MADLAD-400; IC3-v2 states no licence). The IGC restricted-licence parts (News2, Books) were not used. MIM-GOLD-NER, used only for evaluation, has a research-only licence, so no model fine-tuned on it is released.

Model

Fill-mask. Write the mask token right after the previous word, with no space: Reykjavík er höfuðborg[MASK]. The tokenizer keeps each word's leading space inside the word's token, so a space typed before the mask becomes a token of its own. In training such a lone space comes mostly before a number, because digits are split off, so with the space the model predicts digits.

Training

Data

Deduplicated (exact + MinHash 0.75 across all sources, curated sources kept first), quality- and language-filtered, and decontaminated against every evaluation set (13-gram and whole-sentence matching). Per-source epochs below are for the 1,024-token phase.

Sources: IGC-2024 open part (arnastofnun/IGC-2024, CC BY 4.0); the non-IGC subsets of danish-foundation-models/icelandic-dynaword (government gazette and Rafbókavefur under Icelandic copyright law's exemptions; Wikipedia, Wikisource, Wikibooks, wiki comments, radio talks CC BY-SA; sagas CC BY; Gutenberg); Icelandic Wikipedia (wikimedia/wikipedia 20231101.is); MaCoCu-is 2.0 (CC0); FineWeb-2 and FinePDFs isl_Latn (ODC-By); HPLT 3.0 isl_Latn; MADLAD-400 is, clean split (ODC-By); IC3-v2 (Miðeind; licence not stated, Common Crawl derived). CulturaX is (uonlp/CulturaX; mC4 + OSCAR, terms accepted 7 Oct; only the 0.34B tokens not already in the other sources). IcePaHC excluded (evaluation source); restricted IGC parts (News2, Books) not used.

Results

Held-out MLM (15% masking) loss per source at the end of training: {"culturax": 1.229, "dynaword": 0.542, "finepdfs": 1.369, "fineweb2": 1.252, "hplt3": 1.299, "ic3v2": 0.893, "igc": 0.853, "macocu": 1.148, "madlad": 1.007, "wiki": 0.931}.

Fine-tuning (same protocol for every model: learning rate picked on dev from a 3-value grid with seed 0, then 5 seeds at that rate; best-dev epoch; 95% CI from a hierarchical bootstrap over seeds and test items):

160 vs 80 tokens/parameter (same size; paired over the same test-item draws; x100):

Tasks: NER = MIM-GOLD-NER (IceBERT split, entity micro F1); POS = UD Icelandic-Modern + GC (UPOS accuracy); ScaLA = acceptability built from the same UD splits (MCC); IceEC = error vs no error (MCC); QA = NQiI + RUQuAD-1 (token F1).

Caveats

• One pretraining seed per size; differences between sizes include pretraining-seed noise that the CIs do not cover.
• The 8x Chinchilla budget is a decoder-derived heuristic; no compute-optimal law for MLM encoders is established.
• Larger sizes repeat the data several times (see epochs); returns from repetition fall off beyond about 4 epochs (Muennighoff et al. 2023).
• IGC JSONL documents were not used for the 8,192-token phase because their paragraph order is not guaranteed.
• AdamW with gradient clipping replaces ModernBERT's StableAdamW; global RoPE theta is 160k from the start.
• The fine-tuning learning rate comes from a 3-value grid per size class. For large only, the grid was extended upward on NER, POS and QA, because the chosen rate had sat at the top edge. The extension used the same seeds and the same dev-based selection (7 Oct). POS moved to 8e-5 (95.3 -> 95.5) and QA moved to 6e-5 (55.6 -> 56.0); NER was unchanged. The baselines kept their 3-value grids.
• Fine-tuning used bf16 autocast; ScandiBERT gave non-finite losses in bf16 and was fine-tuned in fp32 (flagged per run).

Full numbers: the paper, arXiv:2610.36230. The fine-tuning results above are the first round; the paper extends every learning-rate grid until the chosen rate is not at an edge, so its numbers differ slightly. Export parity check (ours vs transformers logits): see export_check.json.

Citation

@misc{einarsson2026icebertopendatamodernbertencoders,
  title         = {Beyond {IceBERT}: Open-Data {ModernBERT} Encoders for {Icelandic} from {12M} to {1B} Parameters},
  author        = {Hafsteinn Einarsson},
  year          = {2026},
  eprint        = {2610.36230},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.36230}
}