Consumer Complaint Classifier (DistilBERT)

distilbert-base-uncased fine-tuned to route a customer's written complaint to one of 5 product categories:

credit_card · credit_reporting · debt_collection · mortgages_and_loans · retail_banking

• Live demo: mai-data-ai/complaint-classifier
• Code and notebook: GitHub

Results (held-out test set, 12,430 complaints)

Per-class results:

Comparison with models trained from scratch (same split)

The SimpleRNN, LSTM and GRU baselines are Keras models (128-d embedding, 128 units, ~3.9M parameters) trained on the same train/validation/test split. Scores for the from-scratch models vary by about ±0.5 points between training runs; the ranking (DistilBERT > GRU ≈ LSTM > SimpleRNN) was the same in every run.

Training

• Data: Kaggle Consumer Complaints Dataset for NLP. 162,421 rows reduced to 124,300 complaints after removing empty narratives, exact duplicates and texts with conflicting labels.
• Split: stratified 80/10/10 (99,440 train / 12,430 validation / 12,430 test), seed 42.
• Class imbalance: credit_reporting is ~45% of the data, so training used class-weighted cross-entropy (weights computed from the training set only).
• Hyperparameters: learning rate 2e-5, weight decay 0.01, warmup 900 steps, batch size 32, 3 epochs, fp16, max length 256. The best checkpoint was selected by validation macro-F1.

Usage

The model expects text cleaned the same way as the training data: lowercase, no punctuation or numbers, no stop words, lemmatized.

import re, nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from transformers import pipeline

nltk.download("stopwords"); nltk.download("wordnet")
STOP = set(stopwords.words("english"))
lemm = WordNetLemmatizer()

def clean_text(text):
    text = re.sub(r"[^a-z\s]", " ", str(text).lower())
    return " ".join(lemm.lemmatize(w) for w in text.split() if w not in STOP)

clf = pipeline("text-classification",
               model="mai-data-ai/complaint-distilbert", top_k=None)

complaint = "A debt collector keeps calling me about a loan I already paid off."
print(clf(clean_text(complaint), truncation=True, max_length=256))

Limitations

• Categories overlap. Most of the 1,725 test errors involve credit_reporting and debt_collection (credit_reporting → debt_collection: 325, debt_collection → credit_reporting: 295).
• Trained on English complaint text from a single source (US consumer complaints), so performance on other domains or languages is not guaranteed.
• Only 5 coarse product categories; it does not predict issue type or sub-product.
• Intended for assisting complaint routing, not for fully automated decisions without human review.