mmBERT Multilingual Spam Detector

A multilingual spam detection model fine-tuned on jhu-clsp/mmBERT-small using the agentlans/common-crawl-spam dataset.

• Model Architecture: ModernBERT
• Problem Type: Regression (Single-output spam scoring)
• Number of Labels: 1
• Vocabulary Size: 256,000
• License: MIT

Quick Start & Usage

You can easily use this model for inference with the Hugging Face pipeline:

import torch
from transformers import pipeline

model_name = "agentlans/mmBERT-small-spam-detector"

regressor = pipeline(
    "text-classification",
    model=model_name,
    tokenizer=model_name,
    device=0 if torch.cuda.is_available() else -1,
    function_to_apply="none",
)

messages = [
    "The meeting starts at 10 AM tomorrow.",
    "WIN $$$ NOW!!! FREE MONEY GUARANTEED!!! CLICK HERE!!!",
]

predictions = regressor(messages, batch_size=16, truncation=True)

for message, prediction in zip(messages, predictions):
    score = prediction["score"] # logits for the "spam" label prediction
    print(f"Message: {message}\nPredicted spam score: {score:.3f}\n")

Evaluation & Performance

Test Set Performance

• ROC-AUC: 0.9327 (Evaluated on n = 10,000 test samples from agentlans/common-crawl-spam).
• Decision Rule: In this dataset, a sample is classified as spam if its logit > 0. Points along the ROC curve represent alternative cutoff thresholds for the classifier's output.

Example Predictions & Qualitative Analysis

The dot plot below illustrates predicted spam scores across various categories and languages, highlighting the expected progression from informational messages to pure spam. Note the score variability observed within the three least spam-like categories.

The raw sample classification results are in the following table:

Click here for Table 1

Training Details

Click here for training details

Hyperparameters

• Learning Rate: 5e-05

• Train Batch Size: 8

• Eval Batch Size: 8

• Optimizer: AdamW (Fused)

• Number of Epochs: 3.0

• Mixed Precision: BF16

• Gradient Accumulation Steps: 1

• Learning Rate Scheduler: Linear

• Warmup Steps: 0

• Warmup Ratio: 0.0

• Weight Decay: 0.0

• Max Gradient Norm: 1.0

• Number of GPUs: 1

• Seed: 42

Training Results & Evaluation Metrics

Speed Performance

• Training Runtime: 1,302.34 seconds
• Train Samples per Second: 184.28
• Evaluation Runtime: 10.82 seconds
• Eval Samples per Second: 924.47

Framework Versions

• Transformers: 5.17.0
• PyTorch: 2.13.0+cu130

Limitations & Bias

• Inherits biases and distributions present in the training dataset, which primarily focuses on webpage spam and features a strong English-language skew.
• May not generalize across all niche or emerging spam vectors.
• Users should carefully evaluate the downstream impacts of false positives and false negatives for their specific deployment use case and adjust classification thresholds accordingly.
• While ModernBERT offers efficiency gains over standard encoder architectures, transformer-based text scoring can have higher latency compared to lightweight linear/bag-of-words baselines.

License

This model is licensed under the MIT License