flan-t5-base-text-paraphraser
Text paraphraser: google/flan-t5-base fine-tuned (full fine-tuning, standard seq2seq cross-entropy) on the
paranmt5m configuration of redis/langcache-sentencepairs-v3 (ParaNMT-5M sentence pairs). No other data was used. Training used a seeded random subset of 1,000,000 pairs out of 5,041,448 usable training pairs.
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "thealper2/flan-t5-base-text-paraphraser"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
text = "paraphrase: the weather was extremely cold yesterday ."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, num_beams=8, num_return_sequences=3, max_new_tokens=128)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
Input format: paraphrase: <sentence> (prefix is required; also stored in paraphraser_config.json).
Default generation config: beam search, num_beams=4, max_new_tokens=128.
Training data
- Source:
redis/langcache-sentencepairs-v3, configparanmt5m(singletrainsplit, alllabel=1). - Direction:
sentence1→sentence2. - Text is lowercased and PTB-tokenized in the source corpus; it was not re-cased or detokenized.
- Cleaning: null/empty removal, whitespace normalization, PTB quote tokens (
'') mapped to"(the T5 vocabulary has no backtick), control-character / U+FFFD / no-letter filtering, removal of pairs identical after normalization or differing only in case/punctuation, exact pair deduplication, word-length-ratio filter (>3.0 when the longer side has ≥10 words), token-length filter (128 source incl. prefix / 128 target). No truncation. - Split: deterministic, grouped by normalized source sentence (seed 42); validation/test pairs sharing any normalized sentence with train were moved to train, test pairs sharing a sentence with validation were dropped. Exact pair leakage: none; sentence-level (loose) leakage: none.
| Stage | Pairs |
|---|---|
Raw paranmt5m train split |
5,370,126 |
| − non_positive_label | 0 |
| − null_values | 0 |
| − non_string_values | 0 |
| − empty_strings | 0 |
| − malformed_replacement_character | 127 |
| − malformed_control_characters | 3,012 |
| − malformed_no_letters | 399 |
| − source_equals_target_after_normalization | 876 |
| − trivial_variant_case_or_punctuation_only | 68,851 |
| − exact_duplicate_pairs | 157,615 |
| − exceeds_token_limits_128_128 | 319 |
| − word_length_ratio_above_3.0 | 17,356 |
| − test pairs overlapping validation (dropped) | 18 |
| Usable pairs | 5,121,553 |
| Split | Pairs | Share |
|---|---|---|
| train | 5,041,448 | 98.44% |
| validation | 39,837 | 0.78% |
| test | 40,268 | 0.79% |
Training procedure
| Setting | Value |
|---|---|
| Base model | google/flan-t5-base (247,577,856 parameters) |
| Training examples | 1,000,000 |
| Epochs | 2.00 |
| Optimizer steps | 62,500 |
| Per-device batch × grad. accumulation | 32 × 1 |
| Effective batch size | 32 |
| Learning rate / schedule | 3e-05 / linear, warmup ratio 0.05 |
| Weight decay | 0.01 |
| Optimizer | adamw_torch_fused |
| Precision | bf16 (TF32 on) |
| Max source / target tokens | 128 / 128 |
| Checkpoint selection | best eval_ibleu = 14.8031 |
| Hardware | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |
| Peak GPU memory (allocated) | 12.36 GB |
| Training time | 3.42 h |
Evaluation
Generation with beam search (num_beams=4); held-out splits of the processed data. BLEU: sacreBLEU
corpus BLEU (13a). ROUGE: rouge_score F-measure, no stemming. BERTScore: roberta-large,
baseline-rescaled. Copy rate: output equal to the input after NFKC/whitespace normalization. Distinct-n: corpus
unique/total n-grams. iBLEU = 0.8·BLEU(ref) − 0.2·BLEU(source). The copy-source baseline outputs the input unchanged.
Evaluation metrics not found.
Examples (random test samples)
Qualitative examples not found.
Limitations
- Trained on lowercased, PTB-tokenized text; outputs follow that format (lowercase, spaces around punctuation).
- ParaNMT pairs were produced by back-translation; some references are loose or change meaning, and the model inherits this noise.
- Sentence-level model: inputs up to 128 tokens; training sentences have at most ~30 words.
- English only. Not evaluated for factual consistency on long or domain-specific text.
- Automatic metrics against a single reference underestimate valid paraphrases.
Citation
ParaNMT-50M: Wieting & Gimpel, 2018, ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. FLAN-T5: Chung et al., 2022, Scaling Instruction-Finetuned Language Models.