Chat Log PII Detector

Finds personal details and secrets in workplace chat lines: PERSON, EMAIL, PHONE, API_KEY, IP_ADDRESS and ID_NUMBER.

Every name, number and sentence it was trained on is made up. Nothing comes from a real person, customer or business.

Related tool: pii-pattern-scanner (a Space that runs in your browser).

Why it exists

Team chat is valuable to AI labs and full of names, phone numbers and the odd pasted key. This model flags those spots so a person can review them before an export is shared.

Use it

from transformers import pipeline

tagger = pipeline("token-classification", model="aitrainingdatabroker/chat-log-pii-detector", aggregation_strategy="simple")
for hit in tagger("[10:42] Bram Okafor: use the key demo_key_a1b2c3d4e5f6g7h8i9j0 on 10.4.8.21 and call (555) 019-2231."):
    print(hit["entity_group"], hit["word"], round(hit["score"], 3))

How it was trained

Fine-tuned from distilroberta-base for 8 epochs on one RTX 3070. Training data: 360 rows from synthetic-chat-logs-pii-spans, all invented, with test rows built from templates the model never saw in training. The scripts that built the data and trained the model are in source/; the full scores are in scores.json.

Test scores

Exact entity match on 140 held-out rows: precision 1.0, recall 1.0, F1 1.0.

Type	Precision	Recall	F1	Count
`API_KEY`	1.0	1.0	1.0	35
`EMAIL`	1.0	1.0	1.0	70
`ID_NUMBER`	1.0	1.0	1.0	35
`IP_ADDRESS`	1.0	1.0	1.0	35
`PERSON`	1.0	1.0	1.0	175
`PHONE`	1.0	1.0	1.0	35

Limits

Trained only on invented text from templates, with test sentences built from templates it never saw. Real text varies far more.
Short single lines. Real keys come in many formats and a pattern scanner should run beside any model.
A score of 1.0 on generated text says little about real text: the test sentences are new templates, but they are written in the same plain style. Test it on a labelled sample of your own.
This model is not legal advice and not a certification. A person should review its output before anything is shared or decided, and your counsel decides what a rule requires.

License and credit

Apache 2.0. Please credit AI Training Data Broker and link to https://aitrainingdatabroker.com/ when you share or build on it.

About

AI Training Data Broker is a broker and technical advisor for companies that sell or license their data to AI labs. It guides the technical side, including export, format, de-identification and compliance, and publishes free guides, checklists and tools.

aitrainingdatabroker/chat-log-pii-detector