LoRA adapter for Qwen3-14B that classifies an email as phishing or legitimate (one word).
GGUF for Ollama: ellachang/phishing-detector-14b-v2-gguf
Use the plain-text prompt below, not the chat template. The model was trained on raw text. Wrapped in Qwen3's chat template it often starts with
<think>and never answers; on the test set this drops accuracy from 100% to 77.5%.
Classify the following email as phishing or legitimate. Reply with a single word: phishing or legitimate.
### Email:
{email}
### Classification:
The answer is the next token(s) after ### Classification:\n, followed by <|im_end|>.
Test set: 267 emails that do not appear in the training data (see Data below). Greedy decoding, Ollama Q4_K_M.
| Accuracy | Human phishing | Human legitimate | LLM phishing | LLM legitimate | |
|---|---|---|---|---|---|
| Qwen3-14B zero-shot | 42.3% | 13/13 | 0/54 | 100/100 | 0/100 |
| this model, chat template | 77.5% | 11/13 | 53/54 | 97/100 | 46/100 |
| this model, raw prompt | 100% | 13/13 | 54/54 | 100/100 | 100/100 |
Caveats:
Human-LLM Generated Phishing-Legitimate Emails (Kaggle), 4 classes: human/LLM × phishing/legitimate. The original stratified 80/10/10 split had about a third of val and test duplicated in train. This version is trained on deduplicated splits: train 2,698 / val 244 / test 267, where test and val keep only emails that were never in any training split.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("unsloth/qwen3-14b-unsloth-bnb-4bit", device_map="auto")
model = PeftModel.from_pretrained(base, "ellachang/phishing-detector-14b-v2-lora")
tok = AutoTokenizer.from_pretrained("ellachang/phishing-detector-14b-v2-lora")
prompt = ("Classify the following email as phishing or legitimate. Reply with a single word: "
"phishing or legitimate.\n\n### Email:\n{}\n\n### Classification:\n")
inputs = tok(prompt.format("<your email here>"), return_tensors="pt").to(model.device) # no chat template
out = model.generate(**inputs, max_new_tokens=4, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
unsloth/qwen3-14b-unsloth-bnb-4bit