slm-125m-ppo
125M-parameter legal/financial model,
preference-optimized with PPO on top of
prajwal55/slm-125m-sft.
This model did not learn to abstain. Its reward rose 3.6x during training and abstention on held-out unanswerable questions stayed at exactly 0 of 121. Numeric fidelity fell from 98.1% to 93.5%.
It is published as the negative half of a controlled comparison. A reward curve that climbs while the target metric does not move is the whole point.
What this phase was trying to fix
The supervised model answers questions its passage does not support — it invents a plausible fact instead of declining. That failure has no gradient in supervised fine-tuning, because the training data contains 8,000 examples of answering and none of declining.
Two axes only: abstention (decline when unsupported) and faithfulness (when supported, keep every claim traceable to a span).
Results
All checkpoints below were trained from the same SFT policy on the same 1,210 frozen preference triples, so the only variable is the algorithm and its hyperparameters. Held-out set is 121 unanswerable prompts, carved before any training and disjoint by source passage. Greedy decoding.
| Metric | SFT baseline | DPO beta=0.1, 1 epoch | DPO beta=0.1, 2 epochs | DPO beta=0.3 | DPO beta=0.5 | PPO |
|---|---|---|---|---|---|---|
| Abstention (held-out, 121) | 0.0% | 60.3% | 66.1% | 0.0% | 0.0% | 0.0% |
| False abstention | 0.0% | 26.3% | 34.7% | 0.0% | 0.0% | 0.0% |
| Token F1 (1,003 answerable) | 0.590 | 0.456 | 0.400 | 0.585 | 0.592 | 0.563 |
| Exact match | 11.4% | 6.6% | 4.8% | 11.3% | 11.8% | 11.9% |
| Numeric fidelity | 98.1% | 99.2% | 99.2% | 98.8% | 99.0% | 93.5% |
| KL from SFT | 0.0000 | 0.3159 | 0.6792 | 0.0227 | 0.0166 | 0.0628 |
Abstention alone is gameable — a model that declines everything scores 100%. Read it beside false abstention or not at all.
Why it failed, measured rather than guessed
PPO reinforces what the policy samples. Sampling 4,000 completions on unanswerable questions from the starting policy produced 0 abstentions — the behaviour had a 0.00% base rate, so there was nothing to reinforce.
A pre-flight scored 8,000 on-policy completions with the reward model before the run:
on-policy pool A median -0.349 p99 +3.969
teacher declines median +9.625 p1 +7.624
declines at or below the on-policy p99: 0.0%
Zero overlap. The reward model does encode "decline" correctly — it just places it in a region of text space the policy never enters. PPO climbed ~1.2 points and remained ~6 short. That prediction was written down before the run.
The headline finding
DPO can consume off-policy chosen text; PPO cannot. With a 0.00% base
rate for the target behaviour, DPO was handed decline text and moved likelihood
mass onto it directly. PPO could only reinforce what it sampled, and it never
sampled a decline.
Abstention has a KL threshold, not a smooth trade-off. Nothing below KL ~0.06 from the SFT policy abstains at all; token F1 falls monotonically with KL regardless of which algorithm caused the drift.
Usage
from transformers import AutoTokenizer, LlamaForCausalLM
tok = AutoTokenizer.from_pretrained("prajwal55/slm-125m-ppo")
model = LlamaForCausalLM.from_pretrained("prajwal55/slm-125m-ppo")
prompt = ("<|bos|><|system|>You are a legal and financial assistant. Answer the "
"question using only the passage provided. Be precise and concise."
"<|user|>" + passage + "\n\nQuestion: " + question + "<|assistant|>")
ids = tok(prompt, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=160, do_sample=False,
eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
Resolve stop and pad ids from the tokenizer, not from LlamaConfig
defaults — this tokenizer is bos=0, eos=1, pad=2.
Limitations
- Not usable closed-book. It is a grounded-extraction model over a supplied passage, at 125.8M parameters.
- Does not decline at all, and is slightly worse than the SFT model it started from.
- Trained on ~1,200 preference pairs. Nothing here is a general claim about DPO or PPO — it is a claim about these algorithms meeting a target behaviour with a zero base rate.
- Checkpoint:
/data/checkpoints/ppo/kl0.05/final.