Loading…
Loading…
Reads one text message and says scam or safe.
A fine-tuned DistilBERT (67M parameters) trained on real scam reports from around the world, real everyday SMS, and written safe service messages. It runs on a normal laptop CPU.
Messages must be masked the same way as in training: links → <URL>, phone numbers → <PHONE_NUMBER>, emails → <EMAIL_ADDRESS>, long numbers → <NUMBER>. The masking code ships with this model as text_utils.py.
import importlib.util
from huggingface_hub import hf_hub_download
from transformers import pipeline
repo = "rasheq/scam-shield-distilbert"
# Load the masking code that ships with the model
path = hf_hub_download(repo, "text_utils.py")
spec = importlib.util.spec_from_file_location("text_utils", path)
text_utils = importlib.util.module_from_spec(spec)
spec.loader.exec_module(text_utils)
classify = pipeline("text-classification", model=repo)
message = "Hi Mum, I dropped my phone, this is my new number. Can you send me $300 today?"
print(classify(text_utils.prepare(message)))
# [{'label': 'scam', 'score': 0.91...}]
Labels: safe (0) and scam (1). A scam score above 0.5 means scam. The model was trained on messages with at least 3 words; shorter messages can't be judged reliably.
Tested once on 1,531 real messages that were never used for training or for choosing the best epoch:
| Metric | Score |
|---|---|
| Accuracy | 97.8% |
| Precision | 98.7% |
| Recall (scams caught) | 98.2% (1,047 of 1,066) |
| F1 | 98.4% |
| False alarm rate | 3.0% (14 of 465 safe messages) |
Read this before trusting the numbers. The safe messages in this test set are personal chats only. On a separate hand-made set of 60 harder messages (real-looking bank alerts, long notices, quiet scams), the model got 51 of 60 right and flagged 7 of 28 real service messages as scams.
| Setting | Value |
|---|---|
| Base model | distilbert/distilbert-base-uncased |
| Extra tokens | <URL>, <PHONE_NUMBER>, <EMAIL_ADDRESS>, <NUMBER> |
| Training data | 17,199 messages (8,527 scam, 8,672 safe) |
| Max length | 128 tokens |
| Batch size | 16 |
| Learning rate | 2e-5, 10% warm-up, linear decay, AdamW (weight decay 0.01) |
| Epochs | up to 4, early stop on validation loss; best epoch 2 |
| Hardware | Apple M3 Pro (MPS), about 3 minutes per epoch |
distilbert/distilbert-base-uncased