thealper2/t5-small-commitbench

google-t5/t5-small fine-tuned on Maxscha/commitbench for commit message generation: given a git diff, generate the commit message describing it.

Task format

Text-to-text. The input is a task prefix followed by the raw git diff, the target is the commit message.

generate commit message: <git diff>

Training data

Maxscha/commitbench official splits, used unchanged:

Languages covered by the dataset: Python, JavaScript, PHP, Ruby, Java, Go.

Training configuration

Truncation at these limits (measured on a 50k sample with the T5 tokenizer):

• 0.7% of the diffs exceed 512 source tokens.
• 4.47% of the commit messages exceed 64 target tokens.

Results

• Final training loss: 3.5762
• Best validation loss: 3.2414

Test split (20,000 examples), beam search with num_beams=4:

Per programming language:

ROUGE and BLEU are lexical-overlap metrics. They do not fully capture whether a commit message describes a change correctly, and generic messages can score well.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/t5-small-commitbench"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

diff = open("change.patch").read()
inputs = tokenizer(
    "generate commit message: " + diff,
    max_length=512,
    truncation=True,
    return_tensors="pt",
)
output = model.generate(
    **inputs,
    num_beams=4,
    max_new_tokens=64,
    length_penalty=1.0,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Default generation settings: num_beams=4, max_new_tokens=64, min_new_tokens=0, length_penalty=1.0, no_repeat_ngram_size=3, do_sample=False (deterministic).

Limitations

• CommitBench replaces identifying literals with placeholder tokens: every diff contains <HASH> instead of commit hashes, and 26.5% of the reference messages contain <I> (numbers), <URL> or <EMAIL>. The model therefore also generates these tokens, e.g. Bumped version to <I>.
• The T5 sentencepiece vocabulary does not cover every character used in source code (curly braces, backslashes, angle brackets), so about 2.35% of the input tokens become <unk>. This limits how precisely the model can read a diff.
• Diffs longer than 512 tokens are truncated; the tail of the change is not visible to the model.
• CommitBench splits are random over commits, not over repositories: 98.6% of the test examples come from repositories that also appear in the training split. No (diff, message) pair is shared across splits, but the reported scores partly reflect familiarity with a project's commit style rather than generalization to unseen code.
• The dataset is English-only and covers six languages; behaviour on other languages or on very large multi-file changes is untested.
• CommitBench is released under CC BY-NC 4.0, which restricts commercial use of the data.

Reproducibility

• python: 3.12.3
• torch: 2.11.0+cu128
• transformers: 5.17.0
• datasets: 4.3.0
• tokenizers: 0.23.2
• seed: 42