Icelandic ModernBERT warm-start study: ws-Mswap-base-s0, 80 tokens/parameter
Paper: Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters (arXiv:2610.36230). If you use this model, please cite the paper. The BibTeX entry is at the end of this card.
Question this run answers. Does a warm start still pay off for an Icelandic encoder, from which source model, and how many Icelandic tokens does it save? IceBERT (Snæbjarnarson et al., LREC 2022) found that continuing a multilingual model "can lead to state-of-the-art results for some downstream tasks". Our size ladder uses the Ettin shapes, so every size has an English Ettin encoder of identical shape (and base has mmBERT-base), and a warm start changes only the starting weights: architecture, data, tokenizer, schedule and evaluation stay those of the scratch models. Every arm is trained at 10, 20, 40 and 80 Icelandic tokens per parameter.
This model is a warm start from mmBERT-base (jhu-clsp/mmBERT-base, which saw Icelandic in its mid-training and decay phases) with its vocabulary replaced by ours through Fast Vocabulary Transfer, size base, pretraining seed 0, at 80 tokens per parameter. This is the curve's main run to 80 tokens per parameter. It kept its state at the start of each shorter budget's decay phase, and the branches b10, b20 and b40 of this repo continue from those states.
Release. Released under the MIT licence with the paper; the training corpus is not released, because not every
source states a licence that permits redistribution. In the paper this arm is called mmBERT-transplant (the repo names keep
the code names S, E, Erand, Mswap and Mkeep). The starting weights come from jhu-clsp/mmBERT-base (MIT licence). Parts of the training data require attribution (CC BY 4.0: IGC; CC BY-SA: Wikipedia, Wikisource) or
come from web crawls with mixed rights (IC3-v2 states no licence). MIM-GOLD and MIM-GOLD-NER, used only for evaluation,
are licensed for research use, so no model fine-tuned on them is released.
Model
| Architecture | ModernBERT: pre-norm, GeGLU, no biases, RoPE (theta 160k on global and local layers, the source models' setting), global attention every third layer and a 128-token window in the others |
| Shape | 22 layers, hidden 768, GeGLU inner 1152, 12 heads |
| Parameters | 136.1M = 25.2M embedding + 111.0M non-embedding |
| Tokenizer | Icelandic byte-level BPE, 32,768 tokens (the scratch ladder's tokenizer) |
| Vocabulary transplant | FVT from 256,000 source tokens: 32,763 of 32,768 target tokens transplanted, 2.46 source pieces per token on average, 0 without pieces (mean embedding) |
| Loads with | transformers AutoModelForMaskedLM (model_type modernbert); logits match our trainer exactly (export_check.json) |
Fill-mask. Write the mask token right after the previous word, with no space: Reykjavík er höfuðborg[MASK]. The tokenizer keeps each word's leading space inside the word's token, so a space typed before the mask becomes a token of its own. In training such a lone space comes mostly before a number, because digits are split off, so with the space the model predicts digits.
Training
| Tokens seen | 10.89B tokens = the text of 80 tokens per parameter of the scratch ladder at this size |
| Steps | 10,385 at 1,048,576 tokens per step |
| Objective | MLM, 30% masking, 15% in the decay phase |
| Optimiser | AdamW (0.9, 0.98), eps 1e-6, weight decay 1e-05, clip 1.0, bf16 autocast |
| Schedule | warmup-stable-decay: 327M warmup tokens, peak LR 0.0008 (the scratch LR; a pilot at xs found lower rates worse for warm starts), 1-sqrt decay over the last 15%, last 10% at 8,192 tokens |
| Data | the scratch ladder's corpus: 9.02B unique tokens from nine open sources (IGC-2024 open part, Icelandic Wikipedia, Dynaword non-IGC subsets, MaCoCu, FineWeb-2, FinePDFs, HPLT 3.0, MADLAD-400, IC3-v2), deduplicated, filtered and decontaminated against the evaluation sets |
| Hardware | MareNostrum 5 (4x H100 64 GB per node), DDP, FlexAttention with document masks |
Results
Held-out MLM loss (15% masking) per source at the end of training: {"dynaword": 0.536, "finepdfs": 1.277, "fineweb2": 1.204, "hplt3": 1.249, "ic3v2": 0.825, "igc": 0.829, "macocu": 1.078, "madlad": 0.982, "wiki": 0.664}.
Downstream point of this arm, size and budget (x100). Each point pools the arm's pretraining seeds at this size (so it is not this checkpoint alone at xs, s and m) and 3-5 fine-tuning seeds at the learning rate the 80-point model chose on dev. 95% CIs are a hierarchical bootstrap over pretraining seeds, fine-tuning seeds and test items; differences are paired over the same item draws. "Scratch at 160" is the ladder's model trained on twice the 80-point tokens (it also saw CulturaX's unique remainder, +3.7%).
| task | score [95% CI] | pretraining seeds pooled | fine-tuning runs | minus scratch, same budget | minus scratch at 160 |
|---|---|---|---|---|---|
| QA: NQiI + RUQuAD-1, token F1 | 56.9 [55.3, 58.4] | 1 | 5 | +2.6 [+1.7, +3.4] | +2.4 [+1.3, +3.5] |
| NER: MIM-GOLD-NER (IceBERT split), entity micro F1 | 94.2 [93.4, 95.0] | 1 | 5 | +0.9 [+0.4, +1.4] | +0.6 [+0.1, +1.2] |
| ScaLA-style acceptability from the Icelandic UD treebanks, MCC | 73.0 [69.1, 76.8] | 1 | 5 | +5.5 [+2.1, +8.3] | +3.0 [-0.2, +6.5] |
| IceEC sentence-level error detection, MCC | 49.9 [47.8, 52.3] | 1 | 5 | +2.6 [+0.7, +4.6] | +1.8 [-0.4, +4.1] |
| UD UPOS accuracy | 95.4 [95.0, 95.7] | 1 | 5 | +0.1 [-0.1, +0.2] | +0.0 [-0.1, +0.2] |
IceBERT's benchmarks and the added tasks (v2 protocol: learning-rate grids extended until the choice is interior; five fine-tuning seeds; PoS over the ten MIM-GOLD folds):
| task (metric) | score [95% CI] | IceBERT | minus IceBERT [95% CI] |
|---|---|---|---|
| const (f1) | 90.4 [89.7, 91.1] | 90.5 | -0.2 [-0.6, +0.3] |
| dep_icepahc (las) | 87.1 [86.7, 87.4] | 87.2 | -0.1 [-0.3, +0.0] |
| dep_icepahc_clean (las) | 86.5 [86.1, 87.0] | 86.6 | -0.1 [-0.2, +0.2] |
| dep_modern (las) | 79.2 [78.2, 80.0] | 79.2 | -0.0 [-0.5, +0.5] |
| gedcat (f1) | 36.6 [34.4, 39.0] | 34.2 | +2.4 [+1.0, +4.0] |
| gedcat_macro (macro_f1) | 36.8 [33.8, 40.3] | 32.9 | +3.9 [+1.1, +7.7] |
| gedtok (f1) | 41.2 [38.8, 43.1] | 39.1 | +2.1 [+0.5, +3.6] |
| gedtok_macro (macro_f1) | 69.6 [68.4, 70.6] | 68.5 | +1.0 [+0.1, +1.8] |
| iceec (mcc) | 49.9 [47.6, 52.2] | 47.1 | +2.8 [+0.8, +4.9] |
| iwg (acc) | 53.0 [49.7, 56.6] | 53.8 | -0.8 [-4.3, +2.8] |
| ner (f1) | 94.2 [93.5, 94.9] | 94.1 | +0.1 [-0.4, +0.6] |
| pos (acc) | 95.4 [95.0, 95.7] | 95.4 | -0.1 [-0.3, +0.2] |
| posmim (acc) | 97.8 [97.4, 98.1] | 97.8 | +0.0 [-0.0, +0.1] |
| qa (f1) | 56.9 [55.4, 58.3] | 56.9 | -0.0 [-0.9, +0.8] |
| scala (mcc) | 73.0 [69.4, 76.7] | 73.6 | -0.5 [-3.6, +2.8] |
Caveats
- One pretraining seed at base, large and xl (1B); the CIs cover fine-tuning and test-item noise only.
- Branches reuse the fine-tuning learning rate chosen for the 80-point model of the same arm and size.
- The scratch arm reproduces the original ladder (base: 1.144 against 1.141 validation loss).
- Fine-tuning used bf16 autocast; parsers fall back to fp32 on non-finite losses.
Full numbers: the paper, arXiv:2610.36230.
Citation
@misc{einarsson2026icebertopendatamodernbertencoders,
title = {Beyond {IceBERT}: Open-Data {ModernBERT} Encoders for {Icelandic} from {12M} to {1B} Parameters},
author = {Hafsteinn Einarsson},
year = {2026},
eprint = {2610.36230},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.36230}
}