DistilBERT multilingual — PII token classification (ES/EN)

Fine-tuned distilbert-base-multilingual-cased for detecting personally identifiable information in Spanish and English business documents. It is the learned component of the PII Detection & Redaction Pipeline, where it is combined with a checksum-backed rule engine and Presidio.

Entity types

10 BIO classes: PERSON, EMAIL, PHONE, ADDRESS, DATE_OF_BIRTH, GOV_ID, USERNAME, CREDENTIAL, IP_ADDRESS, DATE_TIME.

CREDIT_CARD and BANK_ACCOUNT are deliberately absent. The training corpus annotates a card issuer but contains no card numbers and no bank account numbers at all, so the model cannot learn them. In the parent pipeline those two classes are handled by the rule engine, where Luhn and the IBAN mod-97 checksum are arithmetic proofs rather than estimates.

Results

Validation (seqeval, entity level, 2,000 held-out samples):

On complete business documents (out-of-distribution, 250 synthetic invoices, contracts, emails, tickets and forms) F1 drops to 0.651, while the rule baseline reaches 0.712 and the full ensemble 0.765. Formal documents carry explicit field labels (Phone:, Name:) that regexes handle well and that the sentence-level training corpus never contained. Use this model as one component of an ensemble, not on its own.

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("FranJCastilloC/distilbert-pii-ner-es-en")
model = AutoModelForTokenClassification.from_pretrained("FranJCastilloC/distilbert-pii-ner-es-en")

For span decoding, sliding windows over long documents, per-span confidence and redaction, use the pipeline in the linked repository.

Training data

ai4privacy/pii-masking-300k, filtered to English and Spanish. The corpus is fully synthetic: no real person's data was used at any point. Character offsets were re-aligned to this model's own tokenisation rather than reusing the corpus's precomputed BIO tags.

Config: 15,057 training windows, batch 16, lr 5e-05, max length 192, 1 epoch.

Limitations

• Spanish and English only.
• Trained on synthetic sentences; real documents bring OCR noise, abbreviations and inconsistent layout that this evaluation does not capture.
• Degrades on document-style input relative to sentence-style input (see above).
• Should not be the only safeguard in a redaction system. The parent pipeline pairs it with deterministic rules, calibrated confidence scores and a human review queue for exactly this reason.