spp-ro-base-rewarm
Control for spp-ro-mt: the Vanilla model at step 45,000 + 5,000 steps with the same fresh LR schedule, no reflections.
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
Training
- Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
- Schedule. 5,000 steps from the Vanilla checkpoint at step 45,000, fresh schedule (peak LR 5e-5, 250 warm-up steps, cosine to 0); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈0.33 billion token positions in this run; bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.
Architecture
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
Results
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 14.81 | +0.03 |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 22.64 | +0.04 |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.31 | +0.03 |
| Held-out 4-way cloze accuracy (1,000 items) | 88.6 % | McNemar p = 0.77 |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 25.8 % | p = 0.17, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 35.8 % | p = 0.034, Holm p = 0.70 |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 19.8 % | p = 0.75, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 45.0 % | p = 0.71, Holm p = 1.00 |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 40.8 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 48.3 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.153 |
Reference model: flaviussteff/spp-ro-base.
- Perplexity, paired per document: Base-Ro-rewarm has a higher loss than Base-Ro on the held-out documents (mean Δloss +0.0020 nats/token [95 % CI +0.0015, +0.0026], higher loss on 57 % of 3,000 documents, sign test p = 2.4e-14, Holm p = 2.2e-13).
- Perplexity, paired per document: Base-Ro-rewarm and Base-Ro do not differ significantly on the flagged held-out documents (mean Δloss +0.0015 nats/token [95 % CI -0.0008, +0.0043], higher loss on 52 % of 206 documents, sign test p = 0.53, Holm p = 0.73).
- Perplexity, paired per document: Base-Ro-rewarm has a higher loss than Base-Ro on the unflagged held-out documents (mean Δloss +0.0021 nats/token [95 % CI +0.0015, +0.0026], higher loss on 57 % of 2,794 documents, sign test p = 1.2e-14, Holm p = 1.2e-13).
- ConstitutionEval-Ro: no difference from the reference survives Holm correction over the 24 comparisons of the family. Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro-base-rewarm")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro-base-rewarm", dtype=torch.float32).eval()
prompt = "În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Limitations
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
Citation
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}