Bulwark Scam Detector
A multilingual scam / spam / legitimate message classifier, specialised for Turkish. Fine-tuned from Horizon-Labs/phishing-scam-detector-base (mmBERT-base, 307M parameters) to fix one specific weakness: false alarms on real Turkish institutional messages (bank, cargo, e-Devlet, operator, OTP notifications).
Try it: in-browser demo (runs on your device, messages are not uploaded) · Code: github.com/SavienWalker/bulwark · Benchmark: SavienWalker/bulwark-bench-tr
🇹🇷 Türkçe özet: Bulwark; SMS, e-posta ve kısa mesajları dolandırıcılık / spam / temiz olarak ayırır. Gerçek Türk kurum mesajlarında mevcut modelin temiz mesajların yarısına verdiği yanlış alarmı %14'e indirir, istenmeyen mesajları yakalama oranı ise %98'de kalır. Diğer dillerdeki temel yeteneği korunmuştur.
Headline result
On a held-out half of real Turkish SMS from a personal inbox (opened once, after all model choices were made):
| Base model (threshold 0.5) | Bulwark (threshold 0.6) | |
|---|---|---|
| False alarm on legitimate messages ↓ | 53% (54/102) | 14% (14/102) |
| Legitimate message labelled fraud ↓ | 26% | 9% |
| Unwanted (spam + fraud) caught ↑ | 100% (52/52) | 98% (51/52) |
| 3-class accuracy ↑ | 64% | 87% |
The base model flags roughly every second real bank / cargo / e-Devlet notification as unwanted because it was trained on synthetic scams plus generic web text, and has rarely seen what a real institutional message looks like. Bulwark keeps the base model's recall while learning the difference.
Labels and usage
| id | label | meaning |
|---|---|---|
| 0 | legitimate |
genuine personal or institutional message |
| 1 | spam |
advertising from a real business |
| 2 | fraud |
tries to steal money, card data, passwords, codes, identity or trust |
Recommended decision rule: treat a message as unwanted when p(spam) + p(fraud) > 0.6
(chosen on the development split; stored in bulwark_config.json).
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "SavienWalker/bulwark-scam-detector"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
text = "PTT: Gonderiniz adres eksikligi nedeniyle bekletilmektedir. 0,99 TL odeme yapin: ptt-gonderitakip.com"
with torch.no_grad():
p = torch.softmax(model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits, -1)[0]
probs = {model.config.id2label[i]: round(v.item(), 3) for i, v in enumerate(p)}
unwanted = probs["spam"] + probs["fraud"] > 0.6
print(probs, "unwanted" if unwanted else "ok")
In the browser / Node.js (transformers.js): onnx/model_fp16.onnx
gives the same decisions as the PyTorch model on all 536 real and hard-set messages (max probability difference
0.008). Load it with dtype: "fp16". Avoid int8 quantization: it changed ~4% of decisions and raised false alarms.
import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";
const tok = await AutoTokenizer.from_pretrained("SavienWalker/bulwark-scam-detector");
const model = await AutoModelForSequenceClassification.from_pretrained("SavienWalker/bulwark-scam-detector", { dtype: "fp16" });
const { logits } = await model(tok(text, { truncation: true, max_length: 256 }));
Evaluation
All numbers compare the base model and Bulwark on the same messages. Sets are described in
bulwark-bench-tr.
Messages marked ambiguous (mostly illegal-betting ads) are excluded from metrics.
| Set | Type | Metric | Base | Bulwark |
|---|---|---|---|---|
| Real SMS, test half (212) | real, held out | false alarm ↓ | 53% | 14% |
| unwanted caught ↑ | 100% | 98% | ||
| Real e-mail (40) | real, development set | false alarm ↓ | 33% | 19% |
| Hard set v2 (60) | synthetic, human-reviewed | fraud caught ↑ | 82% | 95% |
| false alarm ↓ | 25% | 0% | ||
| contrast pairs fully correct ↑ | 9/13 | 12/13 | ||
| Hard set v1 (42) | synthetic, human-reviewed | contrast pairs fully correct ↑ | 5/7 | 7/7 |
| UCI SMS Spam, held-out half (2,787) | real, English | unwanted caught ↑ | 87% | 96% |
| Multilingual probe (54; de, fr, es, pt, ru, ar) | synthetic, easy | 3-class accuracy | 54/54 | 54/54 |
Contrast pairs are a fake message and the genuine message it imitates (e.g. fake vs. real HGS toll notice); a pair counts only if both are classified correctly — the clearest test of whether a model reads the signals (domain, request, tone) rather than the topic.
How to read these numbers honestly:
- The only fully clean real-world number is the SMS test half. The e-mail set was used for error analysis, so it is reported as development data.
- Hard sets and training templates were written by the same author (an LLM, Claude). A style overlap could inflate those scores. They are reported separately and labelled synthetic.
- The SMS test half contains no non-ambiguous real fraud — real fraud in a personal inbox is almost all illegal betting. Fraud recall is therefore measured on the (synthetic) hard sets only.
- English: half of the UCI SMS Spam Collection was used as replay data during training, so the English gain is in-distribution; the honest claim is "no regression", not "better than the base model".
- Other languages: a small, easy probe shows no regression. Hard cases outside Turkish were not measured.
- Seed variance: retraining the same recipe with 3 seeds gives 4–14% false alarms on the real-SMS dev split (mean 10.6 ± 5.4). The released checkpoint was the best of those on dev, so the held-out test number (14%) is the one to quote, and 10–15% is a realistic expectation if you retrain.
Generalisation to unseen scam wording
Suggested on the Hugging Face forum: 4 whole template families (26 templates incl. near-variants: delivery/customs fees, family impersonation / "send me the code", job/task scams + account renting, legal threats) were removed from training, and recall was measured on those families only. Paired over 3 seeds:
| Seen families | Held-out families | Δ per seed | |
|---|---|---|---|
| Recall on those families (synthetic) | 100% | 95.9 ± 0.7% | −3.3 / −4.6 / −4.4 |
| Real-SMS dev false alarms | 10.6 ± 5.4% | 12.5 ± 2.8% | +11.0 / −4.4 / −1.1 (noise) |
Unseen scam wording costs about 4 points of recall, consistently. The base model gets 92% on the same slice. Weak spot: link-free, formal-tone fraud (e.g. a debt-enforcement threat) and "please forward the 6-digit code" account takeovers.
Training
- Base: Horizon-Labs/phishing-scam-detector-base (Apache-2.0), itself built on jhu-clsp/mmBERT-base (MIT).
- Data (~9k messages):
- ~8k Turkish synthetic messages from 190 hand-written templates with slot filling (institution, amount, date, code, look-alike domain). Covers real institutional notifications, real-business ads, link-based phishing, and link-free social engineering. Patterns were drawn from public warnings (KGM, SGK, PTT, EGM) and public complaint narratives; no complaint text was copied.
- ~1k English messages from the UCI SMS Spam Collection (CC BY 4.0) as replay against forgetting; spam vs. fraud for these was taken from the base model's own prediction (distillation).
- Leakage control: every training message was compared with every evaluation message (token-set Jaccard); anything ≥ 0.5 was removed.
- Personalisation balancing: scammers now use leaked personal data (name, card last 4 digits, plate, old address). These details were injected into legitimate and fraudulent training messages at equal rates, so that "the message knows my name" is not learned as a sign of legitimacy.
- Recipe: word embeddings frozen (110M of 307M parameters trained), bf16, AdamW lr 2e-5, 2 epochs, max length 256, class-weighted loss, fraud examples down-sampled to the legitimate count. Runs on a single 6 GB laptop GPU (RTX 4050) in about 6 minutes.
- Model selection: on a development half of the real SMS, split by sender so the same institution never appears in both halves.
Scope and limitations
- One message at a time. Many real scams unfold over a phone call, a long WhatsApp conversation, a fake website or an ad. Bulwark sees only the text it is given; the first message of a slow scam often looks harmless.
- A message containing your real personal details is not proof that it is genuine. Leaked databases make this trivial for scammers.
- Illegal-betting ads are labelled
fraudbut treated as ambiguous in evaluation. - Small real evaluation sets (hundreds, not thousands). Treat the numbers as indicative.
- Not a replacement for a bank's or carrier's own fraud systems. Do not use it to make decisions about people.
License and attribution
Apache-2.0. Derived from Horizon-Labs/phishing-scam-detector-base (Apache-2.0) and jhu-clsp/mmBERT-base (MIT). English replay data: Almeida, T.A. & Gómez Hidalgo, J.M., SMS Spam Collection (UCI, CC BY 4.0).