Korean PII — multilingual-e5-base

Span-level Korean PII detection, fine-tuned from intfloat/multilingual-e5-base (a multilingual XLM-RoBERTa bidirectional encoder). It detects 9 PII categories as character-offset spans and is trained for multi-domain Korean coverage (conversational, news, and a range of document domains).

Open PII Notebook — load the model and redact Korean PII interactively.

Capabilities

Benchmark Results

Evaluated across three domains, exact character-span F1, with deterministic span normalization (see extract_pii below).

KDPII per-class (conversational, in-domain)

Quick Start

Install

pip install "transformers>=4.40" torch safetensors

Load

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

MODEL_ID = "FrameByFrame/korean-pii-e5-base"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForTokenClassification.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
model.eval()
if torch.cuda.is_available():
    model.cuda()

Inference

The model emits per-token BIOES labels. The helper decodes them into character-offset spans and applies light, deterministic span normalization (strips trailing Korean particles / whitespace from a span, e.g. 민수씨 → 민수, 송파구에 → 송파구). The benchmark numbers above include this normalization.

import re

_TRAILING_JOSA = ["이에요","이라고","입니다","이야","이랑","한테","에게","으로","이가","이는",
                  "에서","이고","예요","씨","님","이","가","은","는","을","를","야","아","에","의","랑","께","고"]
_DATE_END = re.compile(r".*(?:일|[0-9])", re.S)

def _normalize(text, label, s, e):
    while s < e and text[s] in " .,\t\n": s += 1
    while e > s and text[e-1] in " .,\t\n": e -= 1
    if label == "private_date":
        m = _DATE_END.match(text[s:e])
        if m and m.end() > 0: e = s + m.end()
    elif label in ("private_person", "personal_handle", "private_address"):
        for _ in range(2):
            seg = text[s:e]
            for j in _TRAILING_JOSA:
                if seg.endswith(j) and (e - s) - len(j) >= 2:
                    e -= len(j); break
            else:
                break
    return s, e

def extract_pii(text: str, max_length: int = 256):
    enc = tokenizer(text, truncation=True, max_length=max_length,
                    return_offsets_mapping=True, return_tensors="pt")
    offsets = enc.pop("offset_mapping")[0].tolist()
    with torch.no_grad():
        logits = model(**{k: v.to(model.device) for k, v in enc.items()}).logits
    pred = logits.argmax(-1)[0].tolist()
    id2label = model.config.id2label

    spans, active = [], None  # active = [label, start, end]
    for i, lid in enumerate(pred):
        label = id2label[int(lid)]
        cs, ce = offsets[i]
        if cs == ce:  # special token
            if active: spans.append(active); active = None
            continue
        if label == "O":
            if active: spans.append(active); active = None
            continue
        prefix, cat = label.split("-", 1)
        if prefix in ("B", "S") or not active or active[0] != cat:
            if active: spans.append(active)
            active = [cat, cs, ce]
        else:
            active[2] = ce
    if active: spans.append(active)

    out = []
    for cat, s, e in spans:
        s, e = _normalize(text, cat, s, e)
        if text[s:e].strip():
            out.append({"label": cat, "start": s, "end": e, "text": text[s:e]})
    return out

Redaction

def redact(text: str) -> str:
    spans = sorted(extract_pii(text), key=lambda s: s["start"], reverse=True)
    for s in spans:
        text = text[:s["start"]] + f"[{s['label'].upper()}]" + text[s["end"]:]
    return text

>>> redact("김민수님의 번호는 010-1234-5678입니다.")
"[PRIVATE_PERSON]님의 번호는 [PRIVATE_PHONE]입니다."

Output Schema

Training Details

Known Limitations

• personal_handle (~0.86 in-domain) is the weakest class — handles are open-vocabulary (arbitrary usernames) and overlap with names; near its practical ceiling.
• Held-out document-domain F1 (0.995) is optimistic — those domains are unseen, but share the generator/entity distribution of the synthetic training data. It shows domain-content transfer, not guaranteed real-world-text robustness. Treat real-world performance as bounded by the KDPII (0.94, real conversational) and KLUE-news (0.87, real news) numbers.
• Evaluate on your own domain before high-stakes use. Coverage is broad but not exhaustive; Korean PII annotation conventions vary by source.
• Structured PII (phone/email/url/ip/account/RRN) is best paired with a regex/checksum validator in production for guaranteed precision.
• The extract_pii helper applies span normalization; if you decode logits yourself, apply equivalent trimming to reproduce the reported numbers.

License

MIT — inherited from the base intfloat/multilingual-e5-base (MIT). Training data includes KDPII (CC BY 4.0).

Citation

@misc{framebyframe-korean-pii-e5-base-2026,
  title  = {Korean PII (multilingual-e5-base): token classification for Korean PII},
  author = {Mariappan, Vijayachandran},
  year   = {2026},
  url    = {https://huggingface.co/FrameByFrame/korean-pii-e5-base}
}

Contact

For inquiries, please contact vijay@artelligence.ai