Consumer Complaint Classifier (DistilBERT)

distilbert-base-uncased fine-tuned to route a customer's written complaint to one of 5 product categories:

credit_card · credit_reporting · debt_collection · mortgages_and_loans · retail_banking

Live demo: mai-data-ai/complaint-classifier
Code and notebook: GitHub

Results (held-out test set, 12,430 complaints)

Metric	Score
Accuracy	0.8612
Macro precision	0.8499
Macro recall	0.8486
Macro F1	0.8491

Per-class results:

Class	Precision	Recall	F1	Support
credit_card	0.8302	0.7902	0.8097	1,497
credit_reporting	0.8917	0.8967	0.8942	5,617
debt_collection	0.7936	0.7871	0.7903	2,100
mortgages_and_loans	0.8555	0.8615	0.8585	1,870
retail_banking	0.8784	0.9071	0.8925	1,346

Comparison with models trained from scratch (same split)

Model	Accuracy	Macro F1
SimpleRNN	0.7973	0.7747
LSTM	0.8191	0.8063
GRU	0.8250	0.8184
DistilBERT (this model)	0.8612	0.8491

The SimpleRNN, LSTM and GRU baselines are Keras models (128-d embedding, 128 units, ~3.9M parameters) trained on the same train/validation/test split. Scores for the from-scratch models vary by about ±0.5 points between training runs; the ranking (DistilBERT > GRU ≈ LSTM > SimpleRNN) was the same in every run.

Training

Data: Kaggle Consumer Complaints Dataset for NLP. 162,421 rows reduced to 124,300 complaints after removing empty narratives, exact duplicates and texts with conflicting labels.
Split: stratified 80/10/10 (99,440 train / 12,430 validation / 12,430 test), seed 42.
Class imbalance: credit_reporting is ~45% of the data, so training used class-weighted cross-entropy (weights computed from the training set only).
Hyperparameters: learning rate 2e-5, weight decay 0.01, warmup 900 steps, batch size 32, 3 epochs, fp16, max length 256. The best checkpoint was selected by validation macro-F1.

Usage

The model expects text cleaned the same way as the training data: lowercase, no punctuation or numbers, no stop words, lemmatized.

import re, nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from transformers import pipeline

nltk.download("stopwords"); nltk.download("wordnet")
STOP = set(stopwords.words("english"))
lemm = WordNetLemmatizer()

def clean_text(text):
    text = re.sub(r"[^a-z\s]", " ", str(text).lower())
    return " ".join(lemm.lemmatize(w) for w in text.split() if w not in STOP)

clf = pipeline("text-classification",
               model="mai-data-ai/complaint-distilbert", top_k=None)

complaint = "A debt collector keeps calling me about a loan I already paid off."
print(clf(clean_text(complaint), truncation=True, max_length=256))

Limitations

Categories overlap. Most of the 1,725 test errors involve credit_reporting and debt_collection (credit_reporting → debt_collection: 325, debt_collection → credit_reporting: 295).
Trained on English complaint text from a single source (US consumer complaints), so performance on other domains or languages is not guaranteed.
Only 5 coarse product categories; it does not predict issue type or sub-product.
Intended for assisting complaint routing, not for fully automated decisions without human review.

Loading…

Results (held-out test set, 12,430 complaints)

Metric	Score
Accuracy	0.8612
Macro precision	0.8499
Macro recall	0.8486
Macro F1	0.8491

Per-class results:

Class	Precision	Recall	F1	Support
credit_card	0.8302	0.7902	0.8097	1,497
credit_reporting	0.8917	0.8967	0.8942	5,617
debt_collection	0.7936	0.7871	0.7903	2,100
mortgages_and_loans	0.8555	0.8615	0.8585	1,870
retail_banking	0.8784	0.9071	0.8925	1,346

Comparison with models trained from scratch (same split)

Model	Accuracy	Macro F1
SimpleRNN	0.7973	0.7747
LSTM	0.8191	0.8063
GRU	0.8250	0.8184
DistilBERT (this model)	0.8612	0.8491

Training

Data: Kaggle Consumer Complaints Dataset for NLP. 162,421 rows reduced to 124,300 complaints after removing empty narratives, exact duplicates and texts with conflicting labels.

Split: stratified 80/10/10 (99,440 train / 12,430 validation / 12,430 test), seed 42.

Class imbalance: credit_reporting is ~45% of the data, so training used class-weighted cross-entropy (weights computed from the training set only).

Hyperparameters: learning rate 2e-5, weight decay 0.01, warmup 900 steps, batch size 32, 3 epochs, fp16, max length 256. The best checkpoint was selected by validation macro-F1.

Usage

The model expects text cleaned the same way as the training data: lowercase, no punctuation or numbers, no stop words, lemmatized.

import re, nltk from nltk.corpus import stopwords from nltk.stem import WordNetLemmatizer from transformers import pipeline nltk.download("stopwords"); nltk.download("wordnet") STOP = set(stopwords.words("english")) lemm = WordNetLemmatizer() def clean_text(text): text = re.sub(r"[^a-z\s]", " ", str(text).lower()) return " ".join(lemm.lemmatize(w) for w in text.split() if w not in STOP) clf = pipeline("text-classification", model="mai-data-ai/complaint-distilbert", top_k=None) complaint = "A debt collector keeps calling me about a loan I already paid off." print(clf(clean_text(complaint), truncation=True, max_length=256))

Limitations

Categories overlap. Most of the 1,725 test errors involve credit_reporting and debt_collection (credit_reporting → debt_collection: 325, debt_collection → credit_reporting: 295).

Trained on English complaint text from a single source (US consumer complaints), so performance on other domains or languages is not guaranteed.

Only 5 coarse product categories; it does not predict issue type or sub-product.

Intended for assisting complaint routing, not for fully automated decisions without human review.