DistilBERT GitHub Issue Classifier (bug / feature / question / support)
Full fine-tune of distilbert-base-uncased (66M params): lr 2e-5, batch 16, 4 epochs, weight decay 0.01, 10% warmup, title+body concatenated and truncated to 256 tokens (the truncation used by the NLBSE'24 competition winners).
Training data: pngwn/github-issues-4class — 1,997 balanced issues (bug/feature/question from NLBSE'24 + support class sourced from maintainer-assigned GitHub labels).
Test-set results (1,997 balanced examples)
| metric | value |
|---|---|
| macro-F1 (4-class) | 0.794 |
| accuracy | 0.794 |
| macro-F1 on bug/feature/question subset | 0.779 |
| F1 bug | 0.787 |
| F1 feature | 0.788 |
| F1 question | 0.710 |
| F1 support | 0.891 |
Trails the smaller SetFit MiniLM model (pngwn/github-issue-classifier-setfit-minilm, 22M params, 0.807 macro-F1) on the same data — consistent with the NLBSE literature that contrastive SetFit few-shot recipes outperform straight fine-tunes at this scale.
Usage
from transformers import AutoModelForSequenceClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("pngwn/github-issue-classifier-distilbert")
model = AutoModelForSequenceClassification.from_pretrained("pngwn/github-issue-classifier-distilbert")
inputs = tok("App crashes when opening settings", return_tensors="pt")
preds = model(**inputs).logits.argmax(dim=-1)
Labels (id → name): 0 bug, 1 feature, 2 question, 3 support.