A LoRA adapter that turns Qwen2.5-1.5B-Instruct into a 1–2 sentence Turkish news summarizer.
It was trained with QLoRA on 4907 article–summary pairs from ahmetyaylalioglu/turkish-news-summarization-collection.
Code, prompts, full evaluation and error analysis: github.com/karabunar/turkish-news-summarization
Whole test split (273 articles), greedy decoding, max_new_tokens=80, 4-bit NF4 base. The table compares the adapter with prompting the same base model.
| ROUGE-1 | ROUGE-2 | ROUGE-L | chrF++ | BERTScore-F1 | |
|---|---|---|---|---|---|
| Zero-shot (base) | 25.24 | 13.48 | 21.57 | 27.92 | 56.89 |
| Few-shot, 3 examples (base) | 28.95 | 17.50 | 25.06 | 28.99 | 59.32 |
| Persona + CoT (base) | 25.80 | 14.92 | 22.02 | 28.42 | 56.63 |
| This adapter | 48.62 | 41.74 | 47.12 | 41.70 | 71.34 |
| First-28-words baseline | 43.61 | 36.18 | 40.07 | 42.92 | 65.87 |
ROUGE uses a Unicode-aware Turkish tokenizer. BERTScore uses dbmdz/bert-base-turkish-cased (layer 9).
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
base = "Qwen/Qwen2.5-1.5B-Instruct"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(model, "karabunar/qwen2.5-1.5b-turkish-news-summarization-qlora")
article = "..." # Turkish news text (training and evaluation clipped it to 2500 characters)
messages = [{"role": "user", "content": f"Aşağıdaki haber metnini 1-2 cümle ile anlaşılır şekilde özetle:\n\n{article}"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=80, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
Keep the adapter unmerged on the 4-bit base, as in the evaluation. On an RTX A4000:
| Setup | ROUGE-L | Decoding speed |
|---|---|---|
| Unmerged adapter on the 4-bit base | 47.1 | ~33 % slower per decoding step |
| Merged into the requantized 4-bit base | 27.1 | full speed |
| Merged into a bf16 base | 45.4 | full speed, but twice the weight memory |
| Base | Qwen2.5-1.5B-Instruct, 4-bit NF4 (double quant), bf16 compute |
| LoRA | r=16, α=32, dropout 0.05, targets q_proj, k_proj, v_proj, o_proj |
| Data | 4907 training-split pairs (text → target_summary), shuffle(seed=42). The prompt is the zero-shot instruction above, and the loss is computed on the summary tokens only |
| Optimization | 3 epochs (918 steps), lr 2e-4 cosine, warmup 5 %, batch 2 × grad-accum 8, paged_adamw_8bit, gradient checkpointing, max_seq_length=2048 |
| Hardware | 1× RTX A4000 (16 GB), about 2 h 45 min |
| Stack | torch 2.2.1, transformers 4.44.2, peft 0.12.0, trl 0.9.6, bitsandbytes 0.43.3 |
The adapter is released under Apache 2.0, matching the base model. The training data is CC BY 4.0 (ahmetyaylalioglu/turkish-news-summarization-collection).
Qwen/Qwen2.5-1.5B-Instruct