📰 T5-small Fine-tuned on News Summarization (Custom 25K Dataset)

This is a custom fine-tuned t5-small model trained for abstractive news summarization. The model was trained on a dataset of 25,000 news articles paired with human-written summaries.

---

🔧 Model Details

• Base Model: t5-small (by Google)
• Architecture Modified: Only 2 encoder and 2 decoder layers trained (others frozen)
• Fine-tuned for: Abstractive summarization of news articles
• Dataset Size: 25K article-summary pairs
• Precision: Mixed precision (fp16)
• Training Epochs: 10

---

🧪 Training Configuration

Using Hugging Face's Seq2SeqTrainingArguments:

training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    save_strategy="epoch",
    logging_strategy="steps",
    learning_rate=2e-4,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=4,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=10,
    predict_with_generate=False,
    generation_max_length=128,
    generation_num_beams=4,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    logging_first_step=True,
    fp16=True,
    gradient_accumulation_steps=3,
    dataloader_num_workers=4,
    report_to=None,
    eval_accumulation_steps=3,
    label_smoothing_factor=0.1,
)

---

📈 Training Progress

---

📊 ROUGE Scores Over Epochs

---

Evaluation Metrics

---

How to Use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("your-username/your-model-name")
model = AutoModelForSeq2SeqLM.from_pretrained("your-username/your-model-name")

text = "Your news article goes here."
inputs = tokenizer("summarize: " + text, return_tensors="pt", max_length=512, truncation=True)
summary_ids = model.generate(inputs.input_ids, max_length=128, num_beams=4)
print(tokenizer.decode(summary_ids[0], skip_special_tokens=True))

---

Notes

The model is lightweight and optimized for lower compute environments due to partial layer training.

Ideal for real-time or edge applications requiring summarization of news, reports, and factual narratives.

The generation_max_length can be tuned during inference for better performance on long articles.

---

📍 Intended Use

📰 News summarization

🧾 Document compression

🧠 Low-resource abstractive summarization tasks