Qwen2.5-1.5B-Instruct with LoRA fine-tuned for news summarisation on the BBC News Summary dataset, as part of NewsSum V2, a leakage-free comparison of extractive baselines, fine-tuned transformers and a small LLM. This is the seed-0 model from the study.
| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum | BERTScore | Words |
|---|---|---|---|---|---|
| 66.7 | 54.6 | 41.8 | 65.5 | 48.6 | 161 |
Across three seeds the mean ROUGE-Lsum is 64.9 ± 0.5. The same model used zero-shot scores 38.3 and writes numbers that are not in the article in 7.8% of cases; after this fine-tuning that falls to 0.8%.
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base), "MHkhan50/newssum-qwen2.5-1.5b-lora-bbc")
n = round(0.455 * len(article.split())) # length budget used in training
msg = (f"Summarise the following news article in about {{n}} words. Write complete sentences "
f"and keep the key facts, names and figures. Reply with the summary only.\n\nArticle:\n{{article}}")
prompt = tok.apply_chat_template([{{"role": "user", "content": msg}}], tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=400, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
BBC News Summary (BBC articles from 2004 and 2005, five topics, one summary per article). Near-duplicate articles (154 of 2,225) were removed before a fixed 70/15/15 split stratified by topic: 1,449 training, 311 validation and 311 test articles. The split is published in the repository. Headlines were removed from the input.
Important: the reference summaries in this dataset are sentences copied from the article (94.8% verbatim) and are about 45% as long as the article. This model therefore learned to select and copy sentences in BBC house style, not to write short abstractive summaries.
LoRA adapter (rank 16, alpha 32, dropout 0.05, all linear layers) trained for 3 epochs with AdamW (learning rate 2e-4, effective batch 16), fp16 base weights on an NVIDIA T4, about 70 minutes. Loss only on summary tokens. The prompt asks for about 0.455 × article length in words.
@misc{hussain2026newssum,
author = {Hussain, Muzammal},
title = {When the Reference Is Extracted: TextRank Outscores Fine-Tuned Transformers and a Small LLM on BBC News Summary},
year = {2026},
doi = {10.5281/zenodo.24046039},
url = {https://github.com/MHKhan50/NewsSum-Bench}
}
Qwen/Qwen2.5-1.5B-Instruct