LongT5 — Dialogue Summarization

A LongT5 (tglobal-base) model fine-tuned to summarize short conversations into a sentence or two. Trained on DialogSum + SAMSum — two-person chats and group messenger threads.

• Base model: google/long-t5-tglobal-base
• Task: Abstractive dialogue summarization (English)
• Max input / output: 512 / 96 tokens
• License: Apache-2.0

🔧 Reproducibility: the pipeline lives in code/ as two runnable notebooks — data_processing.ipynb (clean + EDA + investigation) and train.ipynb (fine-tune + evaluate) — plus code/REPORT.md for the full analysis and decision log.

Quick start

The model was trained with a "summarize: " task prefix — add it at inference too:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "tuanhqv123/longt5-meeting-summarization"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

dialogue = """Person1: Can I help you?
Person2: I'd like to buy a new mobile phone please."""
inputs = tok("summarize: " + dialogue, return_tensors="pt", truncation=True, max_length=512)
ids = model.generate(**inputs, max_new_tokens=96, num_beams=4, no_repeat_ngram_size=3)
print(tok.decode(ids[0], skip_special_tokens=True))

Evaluation

Held-out test set (DialogSum + SAMSum), beam=4, ROUGE with stemming + BERTScore-F1:

ROUGE measures word overlap; BERTScore measures semantic similarity, so its higher value reflects that the summaries are usually correct in meaning even when worded differently.

Per-source breakdown

SAMSum (casual messenger chats) summarizes a bit more cleanly than DialogSum (longer, more structured two-person dialogues).

Training

Data: knkarthick/dialogsum + knkarthick/samsum, cleaned and merged (29,610 rows after cleaning). DialogSum's #Person1# tags are normalized to Person1 so the two sources share a consistent speaker style. Each dataset's original train/val/test split is kept.

Validation ROUGE-L per epoch

Early stopping (patience 2) picked epoch 6 — validation ROUGE-L peaked there, then didn't improve.

Intended use & limitations

• Intended: abstractive summarization of short English conversations / chat threads.
• Limitations: English-only; trained on casual/everyday dialogue, so it may transfer poorly to technical, legal, or very long transcripts. Like all abstractive summarizers it can hallucinate — verify facts before relying on a summary. Inputs beyond 512 tokens are truncated.

Citation

Built on LongT5:

@article{guo2021longt5,
  title={LongT5: Efficient Text-To-Text Transformer for Long Sequences},
  author={Guo, Mandy and Ainslie, Joshua and Uthus, David and Ontanon, Santiago and Ni, Jianmo and Sung, Yun-Hsuan and Yang, Yinfei},
  journal={arXiv preprint arXiv:2112.07916},
  year={2021}
}