spp-ro-filtered
Filtered baseline: the same documents; the 75,000 classifier-flagged ones stay in context but are removed from the loss.
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
Training
- Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
- Schedule. 50,000 steps from scratch (peak LR 1.5e-4, 500 warm-up steps, cosine to 0, weight decay 0.1); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈3.28 billion token positions in this run (~66 h); bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.
Architecture
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
Results
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 14.88 | +0.10 |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 23.73 | +1.13 |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.33 | +0.05 |
| Held-out 4-way cloze accuracy (1,000 items) | 88.5 % | McNemar p = 1.00 |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 24.0 % | p = 0.17, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 35.2 % | p = 0.13, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 20.7 % | p = 0.40, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 44.4 % | p = 0.40, Holm p = 1.00 |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 41.2 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 45.0 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.129 |
Reference model: flaviussteff/spp-ro-base.
- Perplexity, paired per document: Filtered-Ro has a higher loss than Base-Ro on the held-out documents (mean Δloss +0.0062 nats/token [95 % CI +0.0045, +0.0078], higher loss on 55 % of 3,000 documents, sign test p = 5.6e-9, Holm p = 4.4e-8).
- Perplexity, paired per document: Filtered-Ro has a higher loss than Base-Ro on the flagged held-out documents (mean Δloss +0.0495 nats/token [95 % CI +0.0405, +0.0594], higher loss on 86 % of 206 documents, sign test p = 4.2e-27, Holm p = 5.0e-26).
- Perplexity, paired per document: Filtered-Ro has a higher loss than Base-Ro on the unflagged held-out documents (mean Δloss +0.0030 nats/token [95 % CI +0.0015, +0.0046], higher loss on 53 % of 2,794 documents, sign test p = 0.001, Holm p = 0.005).
- ConstitutionEval-Ro: no difference from the reference survives Holm correction over the 24 comparisons of the family. Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro-filtered")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro-filtered", dtype=torch.float32).eval()
prompt = "În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Limitations
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
Citation
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}