spp-ro
Synthetic Persona Pretraining from token zero (SPP{T0}): 10% of the training documents carry an inline constitutional reflection.
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
Training
- Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
- Schedule. 50,000 steps from scratch (peak LR 1.5e-4, 500 warm-up steps, cosine to 0, weight decay 0.1); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈3.28 billion token positions in this run (~66 h); bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.
SPP document mode. c1 ... ck <assistant> r1 ... rn ck+1 ...: the reflection (written by an LLM for this very document, grounded in a six-article Romanian value constitution) is inserted at a sentence boundary; the continuation cannot attend to the reflection (asymmetric attention mask) and its RoPE positions alias back to the insertion point, so the document itself is modelled exactly as in the Vanilla model. The <assistant> token carries no loss. At inference, <assistant> is the marker after which the model was trained to write a reflection (there is no post-training).
Architecture
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
Results
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 14.83 | +0.05 |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 22.72 | +0.12 |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.33 | +0.05 |
| Held-out 4-way cloze accuracy (1,000 items) | 88.3 % | McNemar p = 1.00 |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 24.3 % | p = 0.23, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 33.5 % | p = 0.019, Holm p = 0.43 |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 20.2 % | p = 0.65, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 45.2 % | p = 1.00, Holm p = 1.00 |
ConstitutionEval-Ro, letter, deliberative (<assistant> after the scenario) |
21.0 % | |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 40.8 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 46.2 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.150 |
Reference model: flaviussteff/spp-ro-base.
- Perplexity, paired per document: SPP-Ro has a higher loss than Base-Ro on the held-out documents (mean Δloss +0.0026 nats/token [95 % CI +0.0012, +0.0041], higher loss on 54 % of 3,000 documents, sign test p = 4.0e-5, Holm p = 2.8e-4).
- Perplexity, paired per document: SPP-Ro and Base-Ro do not differ significantly on the flagged held-out documents (mean Δloss +0.0038 nats/token [95 % CI -0.0018, +0.0093], higher loss on 53 % of 206 documents, sign test p = 0.37, Holm p = 0.73).
- Perplexity, paired per document: SPP-Ro has a higher loss than Base-Ro on the unflagged held-out documents (mean Δloss +0.0026 nats/token [95 % CI +0.0010, +0.0041], higher loss on 54 % of 2,794 documents, sign test p = 6.5e-5, Holm p = 3.9e-4).
- ConstitutionEval-Ro: no difference from the reference survives Holm correction over the 24 comparisons of the family. Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro", dtype=torch.float32).eval()
prompt = "<assistant>În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Limitations
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
Citation
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}