ModernBERT-base fine-tuned on Banking77

Classifies a message to a bank's customer support into one of 77 intents (card_arrival, lost_or_stolen_card, top_up_failed, ...). Part of TuneBench, which compares it with a TF-IDF baseline, a LoRA-tuned Qwen3-0.6B and Claude Haiku 5.5 on the same test set.

Try it: https://tunebench-henna.vercel.app (the int8 file below behind FastAPI on a Vercel CPU; the first request after a few idle minutes takes a few seconds).

Results on the Banking77 test split (3,080 messages, 40 per intent)

Served as int8 ONNX behind FastAPI with 2 CPU threads (Darwin arm64, 15 cores): p50 4.8 ms, p95 10.1 ms per request.

int8 accuracy depends on the CPU. The numbers above are on an Apple M5 Pro, where ONNX Runtime runs the int8 matrix multiplies in an ARM-specific fused kernel. On x86 the int8 file scores about 91% (91.2% on an Intel Xeon, 91.3% on the live demo); float32 gives the same answers everywhere. The weights use 7 of their 8 bits on purpose: with full 8-bit weights, ONNX Runtime on x86 CPUs without VNNI overflows 16-bit sums, and the first deployment scored 77%.

This is seed 2 of three; the repo reports the mean and spread over all three.

Training

Full fine-tune of answerdotai/ModernBERT-base (revision 8949b90) with a classification head: learning rate 5e-05, batch 32, up to 10 epochs with early stopping on validation macro-F1 (best epoch 7), 10% linear warmup, weight decay 0.01, bfloat16 mixed precision on an Apple M5 Pro GPU. Validation is a stratified 10% of the Banking77 train split; the test split was used once, after training. Code: https://github.com/olioli9586/tunebench.

Use

from transformers import pipeline

classify = pipeline("text-classification", model="Oliilo123/modernbert-base-banking77")
classify("I still haven't received my new card")

The onnx/ folder holds tokenizer.json and labels.json for serving without PyTorch; see src/tunebench/serve.py in the code repo.

Limits

Banking77 is single-turn English messages about a narrow set of retail-banking topics. The model always answers one of the 77 intents, even for messages that fit none of them; it has no "other" class. The most common confusions are between near-duplicate intents (see the error analysis in the code repo).

Dataset: Banking77 (Casanueva et al., 2020), CC BY 4.0.