flan-t5-small-capitalizer

A specialized fine-tuned version of google/flan-t5-small trained on the agentlans/c4-en-lowercased dataset to restore proper capitalization.

Key Features

• Restores proper noun and sentence capitalization
• Builds on FLAN-T5 small's robust NLP capabilities
• Designed for text normalization tasks

Intended Uses

• Capitalizing lowercased text
• Sentence and proper noun capitalization
• Text normalization

Usage Example

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
model_name = "agentlans/flan-t5-small-capitalizer"
flan_t5_pipeline = pipeline("text2text-generation", model=model_name, device=device)

input_text = "buzzfeed's 360-degree look at the aftermath of california's valley fire has been viewed more than 6 million times. plenty of viewers have been asking how we made it."

output = flan_t5_pipeline(input_text, max_length=1024)
print(output[0]["generated_text"])
# Expected output: Buzzfeed's 360-degree look at the aftermath of California's Valley Fire has been viewed more than 6 million times. Plenty of viewers have been asking how we made it.

Limitations

• Language: English only
• Text Type: Primarily modern prose found on the Internet
• Capitalization Issues:
  • May not capitalize titles correctly
  • Inconsistent capitalization style across texts
  • Difficulty with special terms and abbreviations requiring capitalization
• Input/Output Constraint: Maximum length of 1024 tokens for both input and output

Training and evaluation data

The model was trained on a subset of the C4 dataset's English configuration. This dataset contains 125,000 rows, split into 100,000 for training and 25,000 for validation. Each row includes the original text and its lowercased version. It achieves a final validation loss of 0.1338 after processing 56 941 616 input tokens.

Training hyperparameters

The model was trained using the following key hyperparameters:

• Learning rate: 5e-05
• Batch size: 8
• Number of epochs: 1
• Optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• Learning rate scheduler: Linear
• Maximum source and target length: 1024 tokens

Additional training arguments included bf16 precision, automatic batch size finding, and the use of a sortish sampler.

Training results

Click here for table

Framework versions

• Transformers 4.43.3
• PyTorch 2.3.0+cu121
• Datasets 3.2.0
• Tokenizers 0.19.1