t5-small-samsum-demo

This model is a fine-tuned version of google-t5/t5-small on nyamuda/samsum. It reads a dialogue prefixed with summarize: and writes a short summary. Validation loss after 3 epochs: 3.3302.

ROUGE on the first 120 test dialogues, which were not used for training:

Intended uses & limitations

The model is for short dialogue summaries. It still mixes up names and who did what, so the summary should be checked against the dialogue. These ROUGE numbers are from 120 test dialogues, not the full test split.

Training and evaluation data

Training and validation use nyamuda/samsum. The reported ROUGE uses the first 120 rows of the test split.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.05
• num_epochs: 3.0
• label_smoothing_factor: 0.1

Training results

Framework versions

• Transformers 5.22.1
• Pytorch 2.15.0+cpu
• Datasets 5.1.0
• Tokenizers 0.23.3