Preference-tuned (DPO) version of the SFT'd SmolLM-135M ML-paper research assistant. This is stage 3 of a CPT → SFT → DPO pipeline on a 135M-parameter model.
paperbd/smollm_135M_neuraltxt_v1paperbd/paper_preference_150K-v1 — pairs mined by sampling the SFT model and ranking responses with an LLM judge.| Method | DPO (TRL + Unsloth) |
| beta | 0.1 |
| LoRA rank / alpha | 32 / 32 |
| Effective batch | 128 (8 × grad_accum 16) |
| Max seq / prompt length | 1024 / 768 |
| Learning rate | 2e-4, linear decay |
| Epochs | 3 (2,757 steps) |
| Hardware | 1× RTX 3090, ~6h17m |
Held-out 2% split + diversity on 100 sampled prompts (4 responses × 4 temperatures).
| Metric | SFT baseline | This model (DPO) |
|---|---|---|
| Eval loss | — | 0.457 |
| Reward accuracy (held-out) | 0.50 (chance) | 0.72 |
| Reward margin | — | 1.65 |
| Diversity — EAD | 0.1173 | 0.1193 |
| Diversity — SBERT | 0.2263 | 0.2322 |
| Diversity — Vendi | 2.7327 | 2.7410 |
Takeaways: the model learned the preference (reward accuracy 0.50 → 0.72) while preserving output diversity (no mode collapse — all diversity metrics flat vs the SFT baseline). Training shows mild overfitting (train reward accuracy ~0.85 vs eval 0.72), so 3 epochs is the right length.
PaperResearcher task API from the SFT stage.See dpo/DPO_SmolLM135M (run_dpo.sh, experiments.md, LEARNINGS.md).
model.safetensors 269 MBadapter/adapter_model.safetensors 39 MBcheckpoints/checkpoint-2000/adapter_model.safetensors 39 MBcheckpoints/checkpoint-2500/adapter_model.safetensors 39 MBcheckpoints/checkpoint-2757/adapter_model.safetensors 39 MBcheckpoints/final/adapter_model.safetensors 39 MBcheckpoints/checkpoint-2000/optimizer.pt 20 MBcheckpoints/checkpoint-2500/optimizer.pt 20 MBcheckpoints/checkpoint-2757/optimizer.pt 20 MBcheckpoints/checkpoint-2000/rng_state.pth 14.6 kBcheckpoints/checkpoint-2500/rng_state.pth 14.6 kBcheckpoints/checkpoint-2757/rng_state.pth 14.6 kBcheckpoints/checkpoint-2000/training_args.bin 6.3 kBcheckpoints/checkpoint-2500/training_args.bin 6.3 kBcheckpoints/checkpoint-2757/training_args.bin 6.3 kBcheckpoints/checkpoint-2000/scheduler.pt 1.5 kBcheckpoints/checkpoint-2500/scheduler.pt 1.5 kBcheckpoints/checkpoint-2757/scheduler.pt 1.5 kB有文件?提交种子。
提交种子