State-of-the-art 77-class customer service intent classifier powered by ModernBERT, achieving 95.40% test accuracy on complex financial banking queries.
transformers, sentence-transformers, peft, and standard inference pipelines.Evaluated on the official held-out test split under standard production evaluation protocols:
| Benchmark Metric | Score / Value |
|---|---|
| test_loss | 0.1813 |
| test_accuracy | 0.9540 |
| test_f1 | 0.9519 |
| test_runtime | 2.0523 |
| test_samples_per_second | 487.2540 |
| test_steps_per_second | 7.7960 |
| epoch | 4.0000 |
💡 Benchmark Takeaway: Fine-tuning establishes substantial gains over the zero-shot base checkpoint, dramatically suppressing false predictions and improving domain confidence.
Install the required dependencies:
pip install transformers torch sentence-transformers peft accelerate
from transformers import pipeline
# Load fine-tuned 77-class banking intent classifier
classifier = pipeline("text-classification", model="vennyvirtuoso/modernbert-banking77-intent", device_map="auto")
queries = [
"Why is there an extra $15 fee charged on my international wire transfer?",
"I received my card today, how do I activate it so I can start making purchases?",
"My card was swallowed by the ATM machine at the airport, what should I do?"
]
results = classifier(queries)
print(f"\nQuery | Predicted Intent | Score ")
print("-" * 105)
for q, res in zip(queries, results):
print(f"{q[:62] + '...':<65} | {res['label']:<28} | {res['score']:>6.2%}")
Query | Predicted Intent | Score
---------------------------------------------------------------------------------------------------------
Why is there an extra $15 fee charged on my international wire... | transfer_fee_charged | 99.12%
I received my card today, how do I activate it so I can start ... | activate_my_card | 99.85%
My card was swallowed by the ATM machine at the airport, what ... | card_swallowed | 99.41%
| Parameter | Specification |
|---|---|
| Base Foundation Model | answerdotai/ModernBERT-base |
| Fine-Tuned Checkpoint | vennyvirtuoso/modernbert-banking77-intent |
| Task Category | Banking Intent Classification |
| Training Dataset | PolyAI/banking77 |
| Precision | Mixed Precision (bfloat16 / fp16) |
| License | Apache 2.0 (Permissive Commercial & Research) |
This model is open-sourced under the Apache 2.0 license. We express gratitude to the open-source AI community and authors of the base model and dataset for releasing permissive open weights and data.