BERT Phishing URL Classifier (Teacher)
A fine-tuned bert-base-uncased that classifies a URL as phishing or benign from its text alone. It is the teacher in a three-stage compression pipeline:
BERT-base teacher (110M, 12 layers) ──KD──▶ DistilBERT student (~52M, 4 layers) ──NF4──▶ 4-bit student
this repo _student _student_4bit
acc 89.7% acc 96.0% see card
In this pipeline the distilled student ends up better than the teacher. The temperature-softened teacher outputs most likely acted as a strong regulariser. See the student card for details.
Results
Test split (33,000 URLs, balanced):
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| BERT teacher (this) | 0.8971 | 0.9136 | 0.8763 | 0.8945 |
| DistilBERT student | 0.9601 | 0.9710 | 0.9483 | 0.9595 |
Validation split, per epoch:
| Epoch | Train loss | Val loss | Accuracy | Precision | Recall | F1 | ROC-AUC |
|---|---|---|---|---|---|---|---|
| 1 | 0.3139 | 0.2644 | 0.892 | 0.918 | 0.860 | 0.888 | 0.961 |
| 2 | 0.3096 | 0.2569 | 0.895 | 0.912 | 0.873 | 0.892 | 0.963 |
| 3 | 0.3057 | 0.2569 | 0.895 | 0.918 | 0.867 | 0.892 | 0.963 |
Training
| Data | Akash-Sakala/phishing-site-classification: 154k train / 33k val / 33k test |
| Labels | 0 = benign, 1 = phishing |
| Learning rate | 2e-5, linear schedule, 10% warm-up |
| Batch size | 16 |
| Epochs | 3 (28,875 steps) |
| Optimizer | AdamW (fused), β = (0.9, 0.999), ε = 1e-8 |
| Precision | Native AMP (fp16) |
| Seed | 42 |
| Frameworks | Transformers 5.0.0 · PyTorch 2.10.0 · Datasets 4.0.0 · Tokenizers 0.22.2 |
Usage
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Akash-Sakala/bert-phishing-classifier_teacher"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
urls = ["paypal.com.secure-login.verify-account.ru/webscr", "wikipedia.org/wiki/Phishing"]
inputs = tokenizer(urls, return_tensors="pt", truncation=True, padding=True, max_length=128)
with torch.no_grad():
probs = model(**inputs).logits.softmax(-1)[:, 1]
for u, p in zip(urls, probs):
print(f"{p:.2f} {'PHISHING' if p > 0.5 else 'benign'} {u}")
For deployment, prefer the student. It is smaller, faster and more accurate.
Limitations
- The model sees only the URL string. It doesn't check page content, certificates, domain age or reputation feeds, so treat it as one signal in a broader phishing-defence stack.
- Most URLs in the dataset have no
http(s)://scheme. Strip the scheme before inference to match the training distribution. - Phishing patterns drift over time, so accuracy on new campaigns will be lower than on this static test set.
Related
- Student (KD):
bert-phishing-classifier_student - 4-bit student:
bert-phishing-classifier_student_4bit - Dataset:
phishing-site-classification