TinyTurk v2.7d FineWeb 40M — DPO (HH-RLHF)

Direct Preference Optimization (DPO) variant of stunmuffin/tinyturk-v2.7d-fineweb-40m-sft, aligned on Anthropic HH-RLHF (helpful-base).

Model Description

Base model: SFT model (tinyturk-v2.7d-fineweb-40m-sft)
Alignment dataset: Anthropic/hh-rlhf (helpful-base split)
Preference pairs used: 39,521 train / 2,079 val (after length filtering)
Format: ### Instruction:\n...\n\n### Response:\n...
Algorithm: DPO (Rafailov et al., 2023)
Beta: 0.1
Language: English
License: Apache 2.0
Stage: DPO (stage 2 of the alignment pipeline)

Training

Policy model: SFT (trainable)
Reference model: SFT (frozen)
Epochs: 3
Batch size: 4
LR: 5e-6 (cosine decay, 10% warmup)
Optimizer: AdamW (wd=0.01)
Hardware: NVIDIA GTX 1080 Ti
Training time: ~70 min

Results

Epoch	Train Acc	Val Loss	Val Acc
1	0.639	0.6138	0.668 ✅ best
2	0.773	0.6219	0.653
3	0.848	0.6294	0.647

Best epoch: 1 (val acc 0.668). Epochs 2-3 show overfitting on train (acc 0.85) with decreasing val accuracy.

What DPO Improved

Capability	SFT	DPO
Chosen vs rejected discrimination	0.500	0.668
Sentence fluency	Moderate	Improved
Repetition avoidance	Weak	Improved
Format consistency	Good	Better
Factual accuracy	Poor	Still poor
Instruction-response coherence	Poor	Slightly better

Generation Samples

Prompt: What is Python?

SFT:

Python is a programming language that uses Linux to create and execute computers. It allows you, your team or software developer access the data in the system's memory location without having any trouble with it running on an active keyboard account...

DPO:

Python is an extensional programming language that allows users to create and run a variety of programs. It can be used for streaming, running files on screen or performing code executions; however most popular versions include Virtualization (Visa) software such as WordPress, XML-based animations...

Observation: DPO produces more fluent and structured text than SFT. Factual accuracy remains poor — the 42M parameter budget is the bottleneck.

Key Finding

At 42M parameters, DPO improves preference alignment (0.500 → 0.668) and text fluency but cannot inject world knowledge.

This confirms that alignment techniques (SFT, DPO) operate on the format and preference layer, not on the knowledge layer. Knowledge capacity is determined by parameter count and pretraining corpus.

Usage

Important: Set micro_gate=0.75 to match the trained model.

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_16k.model')

config = TinyTurkV27Config(
    vocab_size=16384, context_length=256,
    d_model=512, n_heads=8, n_layers=8, d_ff=2048,
    micro_gate=0.75,
)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = '### Instruction:\nWhat is Python?\n\n### Response:\n'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(80):
        logits, _ = model(x[:, -256:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        if next_id == 2:
            break
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

Cannot produce factually correct content. 42M parameters are insufficient for world knowledge.
EOS token is not learned. Use repetition_penalty and no_repeat_ngram at inference.
Short context. 256 tokens.
English only.
DPO overfits after 1 epoch. Best checkpoint is epoch 1.

Scientific Context

Stage 2 (DPO) of a 3-stage alignment experiment on a 42M-parameter model:

Stage	Model	Key Result
0	Pretrained (FineWeb 40M)	Below GPT-2 Small on all 6 benchmarks
1	SFT (Dolly)	Learns format, cannot produce content
2	DPO (this model)	Aligns preferences (0.668), still cannot produce content

Conclusion: Alignment techniques (SFT, DPO) operate on format and preference; knowledge requires parameter scale.

Citation

@misc{tinyturk-v27d-fineweb-40m-dpo,
  title  = {TinyTurk v2.7d FineWeb 40M — DPO (HH-RLHF)},
  author = {stunmuffin},
  year   = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-fineweb-40m-dpo}
}

References

Rafailov, R., Sharma, A., Mitchell, E., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.

Base pretrained: stunmuffin/tinyturk-v2.7d-fineweb-40m
SFT variant: stunmuffin/tinyturk-v2.7d-fineweb-40m-sft
Dataset: Anthropic/hh-rlhf

stunmuffin/tinyturk-v2.7d-fineweb-40m-dpo