spp-ro-mt
Midtraining variant (SPP{MT}): the Vanilla model at step 45,000 + 5,000 steps on the annotated stream (67.6% of the stream's documents carry a reflection; for those documents the loss is on the reflection only).
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
Training
- Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
- Schedule. 5,000 steps from the Vanilla checkpoint at step 45,000, fresh schedule (peak LR 5e-5, 250 warm-up steps, cosine to 0); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈0.33 billion token positions in this run; bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.
SPP document mode. c1 ... ck <assistant> r1 ... rn ck+1 ...: the reflection (written by an LLM for this very document, grounded in a six-article Romanian value constitution) is inserted at a sentence boundary; the continuation cannot attend to the reflection (asymmetric attention mask) and its RoPE positions alias back to the insertion point, so the document itself is modelled exactly as in the Vanilla model. The <assistant> token carries no loss. At inference, <assistant> is the marker after which the model was trained to write a reflection (there is no post-training).
Architecture
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
Results
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 15.18 | +0.37 |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 23.38 | +0.74 |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.66 | +0.35 |
| Held-out 4-way cloze accuracy (1,000 items) | 88.7 % | McNemar p = 1.00 |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 28.0 % | p = 0.18, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 37.8 % | p = 0.13, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 20.2 % | p = 0.84, Holm p = 1.00 |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 46.7 % | p = 0.008, Holm p = 0.18 |
ConstitutionEval-Ro, letter, deliberative (<assistant> after the scenario) |
27.3 % | |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 37.9 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 43.3 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.152 |
Reference model: flaviussteff/spp-ro-base-rewarm.
- Perplexity, paired per document: SPP-Ro-MT has a higher loss than Base-Ro-rewarm on the held-out documents (mean Δloss +0.0252 nats/token [95 % CI +0.0245, +0.0259], higher loss on 93 % of 3,000 documents, sign test p = <1e-300, Holm p = <1e-300).
- Perplexity, paired per document: SPP-Ro-MT has a higher loss than Base-Ro-rewarm on the flagged held-out documents (mean Δloss +0.0327 nats/token [95 % CI +0.0301, +0.0353], higher loss on 94 % of 206 documents, sign test p = 1.8e-43, Holm p = 2.4e-42).
- Perplexity, paired per document: SPP-Ro-MT has a higher loss than Base-Ro-rewarm on the unflagged held-out documents (mean Δloss +0.0247 nats/token [95 % CI +0.0239, +0.0254], higher loss on 92 % of 2,794 documents, sign test p = <1e-300, Holm p = <1e-300).
- ConstitutionEval-Ro: no difference from the reference survives Holm correction over the 24 comparisons of the family. Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro-mt")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro-mt", dtype=torch.float32).eval()
prompt = "<assistant>În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Limitations
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
Citation
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}