DistilBERT for Email Spam Detection (CEAS-08)

This model is distilbert-base-uncased fine-tuned to classify emails as legitimate (ham) or spam. It was developed in a university project (course DLBAIPNLP01, IU International University of Applied Sciences) that compares classical and transformer-based spam classifiers.

Training data

CEAS-08 corpus from the curated collection by Champa, Rabbi and Zibran (2024), Zenodo, doi:10.5281/zenodo.8339691, licensed under CC BY 4.0. After normalisation and deduplication, 26,470 emails were used for training, 3,309 for validation and 3,309 for testing (stratified split, seed 42).

Input format

The model expects the subject line and the body joined by a newline and normalised as in the project notebook: lower-cased text, HTML removed, mailing-list tags and anonymisation hashes deleted, and URLs, email addresses and numbers replaced by urltoken, emailtoken and numtoken. Inputs are truncated to 256 tokens.

Training procedure

Learning rate 2e-5 with linear decay, batch size 16, weight decay 0.01, three epochs and mixed precision; the epoch with the highest validation F1 score was retained (seed 42).

Evaluation

Spam is the positive class. The model was trained on CEAS-08 only; Enron and SpamAssassin are external test corpora.

Limitations

The training corpus covers a short period in 2008, and its legitimate emails consist largely of mailing-list discussions. Performance decreases considerably on emails from other sources and periods, mainly because types of spam that are absent from the training data are missed, and automatically generated legitimate messages, such as order confirmations, may be classified as spam. The model is intended for educational and research purposes and not for production use.