t5-efficient-base-grammar-correction

Full fine-tune (all parameters, no adapters) of google/t5-efficient-base for English sentence-level grammatical error correction on agentlans/grammar-correction.

Model

Property	Value
Architecture	T5 encoder-decoder (12 + 12 layers)
d_model / d_ff / heads	768 / 3072 / 12
Parameters	222,903,552
Vocabulary	SentencePiece, 32,128 embeddings
Input format	`grammar correction: {sentence}`
Max source / target tokens	256 / 256

Training data

Split	Examples	Use
train	100,000	training
validation	2,000	per-epoch model selection (seeded subset)
validation	25,000	final evaluation below

Token length (train, input with prefix): mean 38.23, p99 129; truncated at 256/256: 0.10% sources, 0.03% targets.
Duplicate pairs: 0 (train), 0 (validation); validation inputs present in train: 0.
Pairs with input == output: 0.
Targets containing characters outside the SentencePiece vocabulary (<unk>): 1.31%.
Text is used as-is (no normalization of case, punctuation or contractions).

Training procedure

Hyperparameter	Value
Optimizer	adamw_torch_fused
Learning rate	0.0001
LR schedule	linear, warmup ratio 0.05
Epochs	5.00
Batch size (per device x accumulation)	16 x 2
Effective batch size	32
Optimization steps	15,625
Weight decay	0.01
Label smoothing	0.00
Max grad norm	1.00
Precision	bf16
Gradient checkpointing	no
Seed	42
Hardware	NVIDIA GeForce RTX 5060 Ti
Training time	1.90 h

Evaluation

Checkpoint selected by eval_sari on 2,000 validation examples: epoch 5.00, step 15,625, validation loss 0.6534 (perplexity 1.922).

Metric	Model (n=25,000)	Copy-input baseline
SARI	68.58	58.40
BLEU (sacrebleu)	70.24	63.57
chrF	84.41	82.49
Exact match (%)	9.10	0.00
Output identical to input (%)	7.38	100.00

Validation loss 0.6504, perplexity 1.916. Decoding: beam search, 4 beams.

Over-correction check: 1,000 grammatical validation references fed as inputs; 24.20% were modified (chrF vs. input 98.81).

Metric notes: BLEU and chrF measure overlap with the reference, so copying the input already scores high (see baseline). SARI (Xu et al., 2016) scores kept/added/deleted n-grams relative to the source on lowercased 13a tokens, so casing edits are not reflected. Exact match is case- and punctuation-sensitive. ERRANT F0.5 was not computed.

Generation config

{
  "max_new_tokens": 256,
  "early_stopping": true,
  "do_sample": false,
  "num_beams": 4,
  "pad_token_id": 0,
  "eos_token_id": 1,
  "decoder_start_token_id": 0
}

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/t5-efficient-base-grammar-correction"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id).eval()

text = "She go to school yesterday."
inputs = tokenizer("grammar correction: " + text, return_tensors="pt", truncation=True, max_length=256)
with torch.inference_mode():
    output_ids = model.generate(**inputs)  # uses the bundled generation_config.json
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Limitations

Training pairs are synthetic (C4_200M-derived corruptions filtered by a grammar classifier); error distribution differs from learner or native-speaker text.
English only, sentence/short-paragraph level; inputs longer than 256 tokens are truncated.
The training set contains no already-correct (input == output) pairs, so leaving correct text unchanged is not directly supervised; check the over-correction numbers above.
The T5 SentencePiece vocabulary cannot represent some characters (e.g. {, }, <, ~, many accented letters); they map to <unk> and are dropped from generated text.
Not evaluated on standard GEC benchmarks (CoNLL-2014, BEA-2019).

thealper2/t5-efficient-base-grammar-correction