ko-pii-detector-ax-tokenclf

Korean PII (Personally Identifiable Information) span detector — token classification (BIO tagging) head on top of skt/A.X-Encoder-base.

• Github: id4thomas/ko-pii-detector

Performance

Entity-level (exact span match) on held-out test set (14,294 examples):

Per label (f1, entity-level exact match):

DRIVER_LICENSE/GENERIC_ID are the weakest labels (high format variance, no fixed pattern to key on).

Full precision/recall per label, CRF-head comparison: ko-pii-detector docs/performance.md

Dataset

Trained on BCCard/pii-masking-openpii-finance with additional noise injection (typos, spacing/format variants, obfuscation, partial-value references, distractor IDs, etc. — see docs/noise.md) to make the model robust to real-world noisy input.

18 span labels:

Usage Example

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

# Load tokenizer, model
tokenizer = AutoTokenizer.from_pretrained("id4thomas/ko-pii-detector-ax-tokenclf", use_fast=True)
model = AutoModelForTokenClassification.from_pretrained("id4thomas/ko-pii-detector-ax-tokenclf")
model.eval()

id2label = model.config.id2label

# predict
text = "문의사항은 010-1234-5678 또는 hong@example.com 으로 연락 주세요."
enc = tokenizer(text, return_tensors="pt", return_offsets_mapping=True)
offsets = enc.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    pred_ids = model(**enc).logits.argmax(dim=-1)[0].tolist()

# merge BIO-tagged tokens into entity spans
entities, cur = [], None
for pred_id, (start, end) in zip(pred_ids, offsets):
    if start == end:
        continue
    tag = id2label[pred_id]
    if tag == "O":
        cur = None
        continue
    prefix, _, label = tag.partition("-")
    if prefix == "B" or cur is None or cur["label"] != label:
        cur = {"label": label, "start": start, "end": end}
        entities.append(cur)
    else:
        cur["end"] = end

for ent in entities:
    ent["value"] = text[ent["start"]:ent["end"]]
    print(ent)
# {'label': 'PHONE', 'start': 6, 'end': 19, 'value': '010-1234-5678'}
# {'label': 'EMAIL', 'start': 23, 'end': 39, 'value': 'hong@example.com'}

More Examples (per label)

One example per label from dataset's held-out validataion

Predictions match gold labels exactly for all rows except PASSPORT, where the model adds one false-positive ADDRESS span (항만점, a bank-branch name suffix).

Attribution

Dataset

This model was trained on BCCard/privacy-filter-openpii-masking dataset by BCCard, licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Source: https://huggingface.co/datasets/BCCard/privacy-filter-openpii-masking. The data was filtered/preprocessed for training. Model weights are released under the Apache License 2.0.

Base Model

Base model: A.X-Encoder-base by SKT (Apache License 2.0) — link