phi-redact — compact English PII/PHI redaction model

phi-redact is an 11M-parameter BERT-mini token classifier for detecting PII and PHI in English text. It covers 32 entity types with BIOES labels and is designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not leave the device.

It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained on ~212k documents (v9 mix) spanning clinical notes, government benefits text, student essays, general PII text, and Wikipedia-style prose.

Not a certified de-identification solution. This model assists redaction workflows. It is not certified under HIPAA Expert Determination or Safe Harbor methods. Evaluate it on your own data before relying on it in compliance-sensitive contexts. English only.

---

Current version: v9 (2026-10-21)

The repo root is v9. v8 and v7 remain available under versioned names: phi-redact-v8-int8.onnx, phi-redact-v8.onnx, phi-redact-v8.mlpackage, phi-redact-v8-fp32.mlpackage, phi-redact-v8-fp32.tflite, model-v8.safetensors, config-v8.json, tags-v8.json (and likewise for v7).

What changed in v9 — a data-only revision; the architecture, teacher, and training procedure are unchanged from v8.

• Mined hard negatives. The shipped v8 model scored a pool of 54,380 gold-empty documents from the training corpora; the 1,616 it fired on (plus 15,000 clean documents) joined the mix as negatives.
• Revised synthetic generator (gen_synth_v9.py, 22k notes): segmented government-ID shapes boosted, 18% public-context / prose-address notes (geography rendered unlabeled in public contexts), 15% sparse low-density notes, and invalid-identifier decoys (impossible SSNs, Luhn-invalid cards, 555 numbers) appended to ~50% of notes.
• Gravitee reduced 25k → 15k (loosest span boundaries in the mix).
• SDK default threshold raised 0.5 → 0.6 (phi-redact-sdk 0.3.0): on the golden set 0.6 dominates 0.5 — higher precision and strict F1 at identical recall.

Golden 164-case synthetic redaction eval (rules + neural merge, the production configuration; the evals/synthetic-164/ set):

v9 figures are the shipped configuration: INT8 ONNX under the SDK scorer (score_phi.py) at the SDK 0.3.0 default threshold of 0.6. The PyTorch checkpoint scores 0.975 full-value recall under the Colab gate harness (evaluate_golden.py) — same test set, two harnesses, small scorer differences. INT8 quantization costs no recall here.

The v8 tradeoff, revisited: v8 bought recall with precision (clean texts picked up a flagged span 44% of the time). v9 improves both axes at once — recall 0.958 → 0.974 and precision 0.695 → 0.746, clean-case FP rate halved to 0.200. What did not improve: public-context retention. The model still redacts most geography that the golden set marks as public/keep (retention 0.133, unchanged from v8) — facility cities, state program names, and similar keep-context spans are now about half of all remaining false positives. If your workflow is precision-sensitive beyond this, v7 remains available above.

Standard benchmarks, v9 student (model only, strict span scoring): Learning Agency micro-F1 0.729 (NAME F1 0.778); FewNERD name F1 0.781 (recall 0.838); MultiNERD name F1 0.893 (recall 0.940); WikiANN name F1 0.763 (recall 0.831); govsynth 1.000. No material regression vs v8 (WikiANN −0.011 is the largest move). The v7 tables below are unchanged and remain the Desert Ant comparison baseline.

Nemotron-PII, v9 (measured 2026-10-21): same 2,000-document structured eval set as below (13,134 gold spans, model only, exact span match). v9 micro P 0.870 / R 0.830 / F1 0.849 — identical F1 to v8 (0.849), itself a wash vs v7 (0.860) on this synthetic distribution; v9's gains are on the clinical-note distribution above. Per entity, v9 improves SSN (0.957 vs v8 0.933), PROFESSION (0.517 vs 0.491) and LICENSE (0.908 vs 0.885), and gives back a little on ADDRESS, DEVICE and HEALTHPLAN.

---

Model details

---

Entity types (32)

NAME_PATIENT · NAME_FAMILY · NAME_PROVIDER · DATE · ADDRESS · PHONE · EMAIL · SSN · MRN · HEALTHPLAN · ACCOUNT · IDNUM · LICENSE · EMPLOYER · PROFESSION · AGE · CITY · STATE · COUNTRY · ZIP · URL · IP · DEVICE · ROUTING_NUMBER · IBAN · CREDITCARD · HOSPITAL · FAX · PASSPORT · VIN · TAX_ID · GOVERNMENT_ID

v8 note: TAX_ID and GOVERNMENT_ID were added in v8 (synthetic EIN / USCIS / A-number templates plus the Gravitee US_ITIN mapping). v9 note: segmented government-ID formats (e.g. Medicare-style 1EG4-TE5-MK73) improved — golden GOVERNMENT_ID full-value recall 4/6 and the adversarial set went 0/2 → 1/2 — but GOVERNMENT_ID remains the weakest structured type.

---

Benchmark results (v7)

Strict span-level scoring: a prediction counts only if both the character boundaries and the label match exactly.

Standard benchmarks

Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.

†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%, within −0.6 pt of Desert Ant's 94.9% on the same dataset.

Nemotron-PII structured benchmark (CC-BY 4.0)

First valid run — 13,134 entities across 25 entity types in 2,000 test documents. MICRO F1 = 0.860, Recall = 0.843.

Selected per-entity F1 scores:

Known weak spot — PROFESSION (F1=0.494, Precision=0.413): the model over-fires on job-title nouns in non-PII context ("She saw a doctor" vs "Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where PROFESSION is rarely the sensitive entity, but flag it if your data is occupation-heavy.

URL in informal prose: Nemotron URL F1=0.828 (formal text) but LA essays URL recall=0.318 (informal prose). The model handles URLs well in structured documents; it misses them in casual first-person writing.

Comparison to Desert Ant Redact v0.4.0 (English)

phi-redact v7 is essentially tied with Desert Ant's English composite at half the parameters and half the on-device footprint.

Inference latency

Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the same 15-template typing-length corpus as evals/latency-bench/bench.py, 15 templates × 10 cycles = 150 redactions, timed from the page's own per-redaction readout — TypeScript rules scan + ONNX detect + merge + placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).

Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).

Caveats: the model was cached for this run; browser version and hardware details were not obtainable from the measurement setup (Linux VM, not a phone); readout resolution is whole milliseconds. Not a head-to-head with any vendor's published number — different hardware, corpus, and setup. Methodology travels with the number; see evals/latency-bench/.

Synthetic redaction eval (164 cases)

evals/synthetic-164/ carries a 164-case synthetic redaction test set, the scoring harness, and v7's results: 0.812 full-value recall (term recall 0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER; weak on NAME (0.347) and ADDRESS (0.333). See the directory README for methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value recall on this set; the 0.812 above is the original 2026-10-02 measurement.)

---

Training data

All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient data was used.

Excluded: ai4privacy/pii-masking-400k (non-commercial license — not used in training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in training corpus).

---

Training procedure

Teacher (distilbert-base-uncased, 66M params):

epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100

Student (this model — BERT-mini, 11.13M params):

epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
distill-alpha 0.7, device A100 SXM4-40GB (~6h)

Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.

v8 (2026-10) used the same procedure on the expanded 205k-document mix. v9 (2026-10) used the same procedure on the revised ~212k-document mix (data-only changes; see the v9 section above).

---

How to use

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact")
model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact")
model.eval()

text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."

enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    logits = model(**enc).logits          # [1, seq, 129]

pred_ids = logits.argmax(-1)[0].tolist()  # [seq]
id2label = model.config.id2label

# Decode BIOES tags to character spans
tokens    = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
offsets   = enc.encodings[0].offsets      # (char_start, char_end) per token

spans = []
span_start, span_label = None, None
for i, tag_id in enumerate(pred_ids):
    tag = id2label[tag_id]
    prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
    label = "-".join(label_parts)
    if prefix == "S":
        spans.append((offsets[i][0], offsets[i][1], label))
    elif prefix == "B":
        span_start, span_label = offsets[i][0], label
    elif prefix == "E" and span_start is not None:
        spans.append((span_start, offsets[i][1], span_label))
        span_start = None
    elif prefix in ("O", "B"):
        span_start = None  # reset on unexpected transition

for start, end, label in spans:
    print(f"[{start}:{end}] {label!r:20s}  {text[start:end]!r}")

For production use, layer the deterministic rules.py (Luhn, ABA routing checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers regardless of model confidence.

---

Limitations

• English only. The bert-base-uncased WordPiece vocabulary has no meaningful cross-lingual representations. French, Spanish, or other languages will produce silent misses — not safe degradation.
• PROFESSION over-fires (F1=0.517 on Nemotron for v9; v7 was 0.494). Treat PROFESSION predictions with lower confidence or disable the entity type if not needed.
• URL in informal prose (recall=0.318 on student essays vs 0.828 on formal text). Check URL recall on your specific document type.
• Over-redaction on clean text. Clean-case FP rate 0.200 on the synthetic golden set at the default threshold (v8: 0.440; v7: 0.120). The remaining mass is public-context geography the golden set marks as keep (retention 0.133) — facility cities and state names in referral/program text. Precision-sensitive workflows can raise the neural threshold or use v7.
• Trained on synthetic data. Real-world typos, OCR noise, and unusual formatting will degrade performance relative to the benchmarks above.
• Not a certified de-identification system. Does not satisfy HIPAA Expert Determination or Safe Harbor. Use as a detection aid alongside human review.

---

License

Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays, WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike requirements apply to the datasets, not the model weights, but consult your legal counsel for commercial deployment.

---

Card generated 2026-09-25 · Updated 2026-10-21 for v9 · Repo: CDT5058/phi-redact