DistilBERT BANKING77 intent classifier

This model classifies short English online-banking support requests into one of the 77 intents defined by BANKING77. It is a learning and research artifact, not a banking decision system.

Results

The original BANKING77 test split was kept out of training and model selection.

The test set is balanced with 40 examples per intent. Aggregate performance hides an important failure: virtual_card_not_working received 0.00 F1. Of its 40 test examples, 19 were confused with getting_virtual_card and 17 with get_disposable_virtual_card. Other common confusions included why_verify_identity versus verify_my_identity, and card_acceptance versus order_physical_card.

Examples from a manual sanity check:

Scores are softmax probabilities, not calibrated confidence estimates.

Training data

• Original dataset: PolyAI/banking77
• Parquet mirror loaded by the training script: DeepPavlov/banking77
• Upstream data license: CC BY 4.0
• Original splits: 10,003 train / 3,080 test
• Working splits: 9,002 train / 1,001 stratified validation / 3,080 untouched test
• Validation seed: 42

The original BANKING77 dataset and authors should be cited. The dataset card does not fully document collection demographics, personal-information review, or social biases.

Training procedure

• Base model: distilbert/distilbert-base-uncased
• Method: full sequence-classification fine-tuning
• Epochs: 5
• Batch size: 32
• Learning rate: 2e-5
• Scheduler: linear with 10% warmup
• Weight decay: 0.01
• Maximum sequence length: 128
• Best-checkpoint metric: validation macro-F1
• Precision: BF16 mixed precision
• Seed: 42
• Hardware: NVIDIA GeForce RTX 5060 Laptop GPU, 8 GiB VRAM
• Transformers: 5.17.0

The run completed 1,410 optimizer steps in approximately 55 seconds. The model was selected using validation macro-F1 and evaluated once on the held-out test split.

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="takuphilchan/banking77-distilbert",
)
print(classifier("My cash withdrawal was declined", top_k=3))

The saved id2label and label2id mappings contain all 77 human-readable intent names.

Intended use and limitations

Intended for demonstrations and research involving English banking-intent classification. It is not suitable for financial advice, account actions, fraud determinations, or autonomous customer support.

Important limitations:

• Inputs outside the 77 known intents are still forced into one of those classes.
• The softmax scores are not calibrated and should not be treated as reliable confidence values.
• Performance is English-only and specific to BANKING77's banking taxonomy.
• Some closely related virtual-card, identity-verification, transfer, and card-ordering intents remain difficult.
• Production use requires current institution-specific data, privacy review, an out-of-domain rejection mechanism, monitoring, and human escalation.
• DistilBERT can inherit biases from its BookCorpus and English Wikipedia pretraining data.

Reproducibility

The training and evaluation code is in scripts/train_classifier.py in the accompanying project. The model was trained with:

uv run python scripts\train_classifier.py `
  --model D:\huggingface-cache\git-distilbert-base-uncased `
  --base-model-id distilbert/distilbert-base-uncased `
  --batch-size 32 `
  --eval-batch-size 128 `
  --output-dir outputs\banking77-distilbert

Detailed per-class metrics and the complete confusion matrix are included in evaluation.json.

Citation

@inproceedings{casanueva-etal-2020-efficient,
  title = {Efficient Intent Detection with Dual Sentence Encoders},
  author = {Casanueva, I{\~n}igo and Tem{\v{c}}inas, Tadas and Gerz, Daniela and
            Henderson, Matthew and Vuli{\'c}, Ivan},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  year = {2020}
}