LFM2.5-350M-grpo-summarization-length-quality-meteor-rouge

⭐ Best run in this series.

Trained with a combined reward: length deviation penalty plus METEOR + ROUGE as quality signal. Both signals were active simultaneously throughout training.

G-Eval Scores (each 0–1; Composite max 4.0)

Faithfulness	Coverage	Conciseness	Clarity	Composite	Pass Rate
0.834	0.493	0.685	0.690	2.701	45.2%

Evaluated on 200 examples from mlabonne/smoltldr test split · judge: gpt-5-mini-2025-08-07 via DeepEval G-Eval (5 rounds averaged, each metric 0–1)

Eval Rollouts

Full per-example outputs, G-Eval scores, significance tests, and summary tables are in the dataset repo: YuvrajSingh9886/reddit-posts-summarization-grpo

Length-Penalty Fine-tuned Runs (alternative strategy)

Run	Faithfulness	Coverage	Conciseness	Clarity	Composite	Pass Rate
`grpo-summarization-quality-meteor` ⭐	0.902	0.569	0.662	0.771	2.904	51.8%
`grpo-summarization-quality-meteor-bleu`	0.901	0.611	0.597	0.769	2.878	46.6%
`grpo-summarization-quality-meteor-rouge`	0.918	0.506	0.704	0.711	2.840	43.3%
`grpo-summarization-quality-bleu`	0.899	0.514	0.603	0.774	2.790	37.9%
`grpo-summarization-quality-bleu-rouge`	0.887	0.275	0.841	0.723	2.726	9.1%
`grpo-summarization-quality-rouge`	0.852	0.262	0.847	0.698	2.659	8.4%

All Length-Penalty Included Runs

Run	Faithfulness	Coverage	Conciseness	Clarity	Composite	Pass Rate
`grpo-summarization-length-quality-meteor-rouge` ⭐	0.834	0.493	0.685	0.690	2.701	45.2%
`grpo-summarization-length-quality-bleu-rouge`	0.696	0.443	0.606	0.643	2.387	35.4%
`grpo-summarization-length-quality-meteor-bleu`	0.696	0.451	0.595	0.634	2.377	34.2%
`grpo-summarization-length-quality-meteor`	0.689	0.433	0.595	0.641	2.358	32.5%
`grpo-summarization-length-quality-rouge`	0.642	0.414	0.575	0.646	2.278	30.1%
`grpo-summarization-length-quality-bleu`	0.620	0.401	0.556	0.665	2.243	26.7%
`grpo-summarization-length-only`	0.627	0.378	0.554	0.674	2.233	24.6%

Usage (MLX)

from mlx_lm import load, generate

model, tokenizer = load("YuvrajSingh9886/LFM2.5-350M-grpo-summarization-length-quality-meteor-rouge")
messages = [
    {"role": "system", "content": "You are an assistant who is an expert at summarization task. The user gives you a post and you are required to summarize it, keeping the key points and main ideas intact, in EXACTLY 50 words"},
    {"role": "user",   "content": "<paste Reddit post here>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=128, verbose=False))

Training Details

Setting	Value
Base model	liquid-ai/LFM-2.5-350M
Algorithm	GRPO (via smolcluster)
Dataset	`mlabonne/smoltldr` (train split, Reddit summarization)
Reward	length penalty + METEOR + ROUGE
Hardware	Apple Silicon Mac mini cluster
Framework	MLX
Weights format	MLX safetensors (bf16)
Eval examples	200 (test split)
Judge	`gpt-5-mini-2025-08-07` via DeepEval G-Eval

Citation

These checkpoints were trained with smolcluster. If you use them in academic work, please cite:

@software{singh2026smolcluster,
  title     = {smolcluster: An Educational Distributed Training and Inference Library for Local Computing},
  author    = {Singh, Yuvraj},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23196836},
  url       = {https://github.com/YuvrajSingh-mist/smolcluster}
}

Released freely — support more experiments like it:

More: https://huggingface.co/YuvrajSingh9886

YuvrajSingh9886/LFM2.5-350M-grpo-summarization-length-quality-meteor-rouge