Ph5 Text Classifier: Social Engineering & Email Phishing Detector (BERT-base)

Model Summary

The Ph5 Text Classifier is a fine-tuned bert-base-uncased language model engineered to detect psychological manipulation, urgency cues, coercive authority claims, and credential-harvesting lures across emails, SMS messages, and HTML message payloads.

It functions as the core semantic inspection branch in the Ph5 Multimodal Cybersecurity Framework, and is directly coupled with an optical character recognition (OCR) fallback engine to analyze text embedded within attached images and QR-code lures.

Performance & Evaluation Benchmark

Evaluated on an independent, strictly isolated held-out test split of 11,652 deduplicated messages (6,747 Benign [57.90%], 4,905 Phishing [42.10%]):

Metric	Measured Score	Standard Deviation (5-Fold CV)
Accuracy	97.22%	± 0.10%
Precision	97.51%	± 0.08%
Recall	95.84%	± 0.12%
F1-Score	96.67%	± 0.09%
ROC-AUC	0.9961	± 0.0004
False Positive Rate	1.78% (FP = 120 / 6747)	—
False Negative Rate	4.16% (FN = 204 / 4905)	—
Mean Inference Time	34.2 ms	Batch Size = 1 (RTX 3050 Laptop GPU)

Baseline Comparison

Architecture	Accuracy	Precision	Recall	F1-Score
TF-IDF + Multinomial Naive Bayes	86.85%	88.32%	79.25%	83.54%
TF-IDF + Logistic Regression	88.27%	89.24%	82.02%	85.48%
Ph5 BERT-base (This Model)	97.22%	97.51%	95.84%	96.67%

Confusion Matrix (Test Split: N = 11,652)

               Predicted
           Benign   Phishing
Benign      6627      120
Phishing     204     4701

Intended Use & Integration

Python (Transformers & PyTorch)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load model and tokenizer from Hugging Face Hub
repo_id = "xynaxdev/ph5-text-bert"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)
model.eval()

# Sample message text
message = "Urgent: Your account credentials have expired. Verify immediately to avoid permanent lockout."

# Tokenize and run forward pass
inputs = tokenizer(message, return_tensors="pt", max_length=256, truncation=True)
with torch.no_grad():
    logits = model(**inputs).logits
    probabilities = torch.softmax(logits, dim=-1).squeeze().tolist()

p_benign, p_phish = probabilities[0], probabilities[1]
print(f"Benign Probability:   {p_benign:.4f}")
print(f"Phishing Probability: {p_phish:.4f}")
# Output: Phishing Probability: 0.9872

Training Configuration & Data Provenance

Pretrained Backbone: bert-base-uncased (12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters)
Classification Head: Linear projection (768 -> 2)
Training Samples: 77,677 messages total (54,374 train / 11,651 val / 11,652 test)
- Sourced from curated open-access corpora on Hugging Face: ealvaradob/phishing-dataset and mrm8488/email-phishing-detection
Optimizer: AdamW (beta1=0.9, beta2=0.999, weight decay = 0.01)
Learning Rate: 2e-5 with linear learning rate warmup
Batch Size: 16 (Trained with PyTorch Automatic Mixed Precision)
Epochs: 3 epochs (Early stopping checkpoint)

Citation & Framework

@article{devi2026ph5,
  title={Ph5: A Multimodal Framework for Cross-Modality Evasion-Resilient Phishing Detection in Multi-Vector Environments},
  author={Ritu Devi, Deepti Singh and Akash Sah},
  journal={Journal of Computer & Mathematical Sciences / JCMM},
  year={2026}
}