spp-ro-base
Vanilla control: the same documents, the same stream, no reflections.
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
Training
- Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
- Schedule. 50,000 steps from scratch (peak LR 1.5e-4, 500 warm-up steps, cosine to 0, weight decay 0.1); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈3.28 billion token positions in this run (~66 h); bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.
Architecture
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
Results
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 14.78 | ref |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 22.60 | ref |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.28 | ref |
| Held-out 4-way cloze accuracy (1,000 items) | 88.4 % | ref |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 27.5 % | ref |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 38.2 % | ref |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 19.5 % | ref |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 45.2 % | ref |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 39.2 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 47.9 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.150 |
- ConstitutionEval-Ro: reference model (0 of the 24 comparisons of the family survive Holm correction). Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro-base")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro-base", dtype=torch.float32).eval()
prompt = "În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Limitations
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
Citation
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}