Multilingual DialogPII NER

A fine-tuned jhu-clsp/mmBERT-base model with a CRF layer for Personally Identifiable Information (PII) detection in multilingual dialogues across 11 languages.

Model Description

This model performs token-level Named Entity Recognition (NER) to identify and classify PII entities in dialogue text. It was trained on synthetic multilingual de-identification of conversational data.

• Architecture: mmBERT-base (ModernBERT) + CRF head with FLERT context windowing
• Training: Fine-tuned on all 11 languages jointly (multilingual training) using FLERT-style document context
• Loss: Cross-Entropy
• Hyperparameters: lr=2e-05, batch_size=32, max_length=2048, dropout=0.1, epochs=10
• Context window: 2 sentences left + 2 sentences right, separated by [SEP] markers
• Decoding: Viterbi decoding via CRF layer

Supported Languages

Entity Types

The model recognizes 19 PII entity types using BIO tagging:

Performance

Evaluated on held-out test sets per language (type-aware micro scores):

Usage

This model uses a custom CRF architecture with FLERT-style context windowing and cannot be loaded directly with AutoModelForTokenClassification. You need to use the custom ModernBertCRF class.

Note: The config.json in this repo exists solely for Hugging Face download tracking. For model loading, use crf_config.json and flert_config.json instead.

Setup

import torch
import json
import re
import torch.nn as nn
import spacy
from transformers import AutoModel, AutoTokenizer
from torchcrf import CRF
from huggingface_hub import snapshot_download

class ModernBertCRF(nn.Module):
    def __init__(self, base_model_name, num_labels, id2label, label2id):
        super().__init__()
        self.num_labels = num_labels
        self.id2label = id2label
        self.label2id = label2id
        self.transformer = AutoModel.from_pretrained(base_model_name)
        hidden_size = self.transformer.config.hidden_size
        self.classifier = nn.Linear(hidden_size, num_labels)
        self.dropout = nn.Dropout(0.1)
        self.crf = CRF(num_labels, batch_first=True)

    def forward(self, input_ids, attention_mask, labels=None, **kwargs):
        kwargs.pop("token_type_ids", None)
        outputs = self.transformer(input_ids=input_ids, attention_mask=attention_mask)
        sequence_output = self.dropout(outputs.last_hidden_state)
        emissions = self.classifier(sequence_output)
        return {"logits": emissions}

    def decode(self, emissions, mask):
        return self.crf.decode(emissions, mask=mask)

# Load model
model_dir = snapshot_download("DFKI-SLT/multilingual_DialogPII_NER")

with open(f"{model_dir}/crf_config.json") as f:
    config = json.load(f)

with open(f"{model_dir}/flert_config.json") as f:
    flert_config = json.load(f)

model = ModernBertCRF(
    base_model_name=config["base_model_name"],
    num_labels=config["num_labels"],
    id2label=config["id2label"],
    label2id=config["label2id"],
)
model.load_state_dict(torch.load(f"{model_dir}/pytorch_model.bin", map_location="cpu"))
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_dir)
id2label = {int(k): v for k, v in config["id2label"].items()}

context_window = flert_config["context_window"]     # 2
use_sep_marker = flert_config["context_sep_marker"]  # True

Preprocessing: Sentence Splitting

The model was trained using FLERT-style context windowing over sentence-level input. Each sentence is predicted with surrounding context sentences. For best results, split your input into sentences using spaCy before inference.

nlp = spacy.blank("en")          # use "de" for German, "xx" for multilingual
nlp.add_pipe("sentencizer")

def split_dialogue(text, nlp):
    sentences = []
    for line in text.strip().splitlines():
        m = re.match(r"^(SPEAKER_\d+)\s*:\s*(.*)", line.strip())
        if m:
            speaker, rest = m.group(1), m.group(2)
            sentences.append([speaker, ":"])
            line = rest
        if not line:
            continue
        doc = nlp(line)
        for sent in doc.sents:
            tokens = [tok.text for tok in sent if not tok.is_space]
            if tokens:
                sentences.append(tokens)
    return sentences

# Example
raw = """SPEAKER_00: Hello, my name is Peter.
SPEAKER_01: Hello, my name is Peter as well. Okay, and where do you come from? I come from Chicago."""

sentences = split_dialogue(raw, nlp)

Inference with FLERT Context Windowing

The key difference from standard token classification: each sentence is predicted within a window of surrounding context sentences, joined by [SEP] tokens. Only labels for the target sentence are extracted.

def predict_dialogue(sentences, model, tokenizer, id2label,
                     context_window=2, use_sep_marker=True, device="cpu"):
    sep = tokenizer.sep_token
    all_labels = []
    for i, target_tokens in enumerate(sentences):
        left  = sentences[max(0, i - context_window):i]
        right = sentences[i + 1:i + 1 + context_window]

        flat_tokens = []
        for s in left:
            flat_tokens.extend(s)
        if use_sep_marker and left:
            flat_tokens.append(sep)

        tgt_start = len(flat_tokens)
        flat_tokens.extend(target_tokens)
        tgt_end = len(flat_tokens)

        if use_sep_marker and right:
            flat_tokens.append(sep)
        for s in right:
            flat_tokens.extend(s)

        enc = tokenizer(flat_tokens, is_split_into_words=True,
                        return_tensors="pt", truncation=False).to(device)
        word_ids = enc.word_ids(batch_index=0)

        with torch.no_grad():
            emissions = model(**enc)["logits"]
            mask = enc["attention_mask"].bool()
            preds = model.decode(emissions, mask)[0]

        word_labels = ["O"] * len(target_tokens)
        seen = set()
        for idx, wid in enumerate(word_ids):
            if wid is None or wid in seen:
                continue
            seen.add(wid)
            if tgt_start <= wid < tgt_end:
                word_labels[wid - tgt_start] = id2label[preds[idx]]

        all_labels.append(word_labels)
    return all_labels


# Run prediction
results = predict_dialogue(sentences, model, tokenizer, id2label,
                           context_window=context_window,
                           use_sep_marker=use_sep_marker)

for sent_tokens, sent_labels in zip(sentences, results):
    for token, label in zip(sent_tokens, sent_labels):
        if label != "O":
            print(f"{token:20s} -> {label}")

Single-sentence inference

For isolated sentences without dialogue context, pass them with context_window=0:

tokens = ["My", "name", "is", "John", "Smith", "and", "I", "live", "in", "Berlin", "."]

results = predict_dialogue([tokens], model, tokenizer, id2label,
                           context_window=0, use_sep_marker=False)

for token, label in zip(tokens, results[0]):
    if label != "O":
        print(f"{token:20s} -> {label}")

Training Data

The model was trained on synthetic multilingual dialogue data covering various domains (medical anamnesis, customer support, police reports, therapy sessions, etc.). The data was generated and annotated as part of a thesis project on multilingual PII de-identification.

Limitations

• Trained on synthetic dialogue data; performance on real-world data may vary
• Optimized for dialogue/conversational text; may underperform on formal documents
• Arabic and Hindi show lower performance compared to European languages
• Requires pytorch-crf package for inference

Citation

If you use this model, please cite:

@misc{roller2026multilingual,
  title={DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information},
  author={Roland Roller and Vera Czehmann and Derya Erman and Luke Flanagan and Ibrahim Baroud and Fr{\'e}d{\'e}ric Blain and Viviana Cotik and Eletta Giusto and Akhil Juneja and Mariana Neves and Maria S{\l}owi{\'n}ska and Christine Hovhannisyan and Aaron Louis Eidt and Lisa Raithel and Sebastian M{\"o}ller and Maija Poikela},
  year={2026},
  institution={DFKI SLT}
}