OpenAI Privacy Filter fine-tune for Brazilian-Portuguese PII

This is a fine-tune of OpenAI Privacy Filter (openai/privacy-filter), not a GLiNER model; the repo name gliner-opf-ptbr-pii-v1 is historical. It was trained on 914,452 rows of natural upstream text in a 9-round chunked schedule (3 epochs × 3 chunks) and evaluated on the same 5,000-row PT-BR val set as the GLiNER family. It has the best partial F1 on that val set (0.897) and by far the fewest false positives on text with no PII (64 on 2,419 rows, vs 8,666 for mmBERT-small), but trails the mmBERT-small GLiNER model on held-out sources. Try it in the demo.

Related repos

Quick start

Needs the opf package and a GPU. The repo root holds the released checkpoint e3_c3 (final round).

from huggingface_hub import snapshot_download
from opf import OPF

path = snapshot_download("arthrod/gliner-opf-ptbr-pii-v1",
                         allow_patterns=["config.json", "model.safetensors"])
runtime = OPF(model=path, device="cuda", output_mode="typed", output_text_only=False)
print(runtime.redact("Meu CPF é 123.456.789-09 e meu telefone é (11) 91234-5678."))

Command line (from USAGE.txt):

opf --checkpoint <checkpoint_dir> --device cuda "Alice was born on 1990-01-02."
opf eval /path/to/eval.jsonl --checkpoint <checkpoint_dir> --device cuda

An intermediate round loads the same way with allow_patterns=["checkpoint-e2_c3/*"] and model=f"{path}/checkpoint-e2_c3".

Files and checkpoint layout

Training data

• Train: arthrod/oai-pf-ptbr-chunked-v2 (private) — 914,452 rows, 100% upstream raw text
  • 99.8% from ai4privacy/open-pii-masking-500k-ai4privacy
  • 100% from ai4privacy/pii-masking-400k
  • 84.6% from arthrod/gliner2-pii-ptbr-reward-split (private, synthetic)
  • 93.4% from nvidia/Nemotron-PII
  • 4 small spam/phishing sources at 100% (negative evidence)
  • 3 sources dropped entirely (schema mismatches, ~18.6k rows)
• Val: same 5,000 PT-BR rows used for the GLiNER models — direct head-to-head comparison
• Chunking: three deterministic chunk files (oai_pf_train_natural_chunk{1,2,3}.jsonl, 304,816–304,818 rows each, 914,452 total), each seen once per epoch (see training_rounds.parquet).

Training recipe

From config.json and the finetune_summary.json files, plus the original run notes:

Evaluation

Headline

opf eval --eval-mode typed on the 5,000-row natural PT-BR val set: detection.span typed F1 = 0.885 (P 0.894, R 0.876) for e3_c3.

Shared PT-BR val benchmark vs the GLiNER family (same val, same 24 labels, nervaluate)

Source: comparison_report.md in this repo (which also has the per-entity partial F1 of every model). GLiNER predictions at threshold 0.5, this model with default Viterbi decoding.

The ettin rows are separate easter-egg fine-tunes, not the public ettin repos of the family. opf-base (untyped) scores 0.000 here because its native vocabulary lacks these labels; its untyped span F1 on the val set is 0.537.

On this in-distribution val set this model wins partial F1 by +0.01 and mmBERT wins exact F1 by +0.017, with different strengths:

opf wins on free-text sensitive descriptors (medical +0.16, organizational +0.34, political +0.24, sexual +0.14, religious +0.04, ethnicity +0.03) mmBERT wins on structured PII + names (first/middle/last names by 0.04–0.16, locations by 0.05–0.14, full address) Both perfect on cpf/rg/pis/credit_card/phone/email/zip (F1 ≥ 0.99)

Held-out sources and text with no PII

The val set above is in-distribution. On held-out sources the ranking changes, and on text with no personal data the two families behave very differently (benchmark dataset quality metrics, GLiNER models at threshold 0.5):

Prefer this model when false positives are costly (a redaction pass a person then reads); prefer mmBERT-small when recall on names and locations matters more.

Learning curve (typed span F1 on the val set, per round)

From the original run notes; plotted in learning_curves.png.

Per-round training metrics (training_rounds.parquet)

Validation loss falls monotonically across the 9 rounds; round 9 (e3_c3, repo root) has the lowest validation loss and the highest validation token accuracy. Source: the finetune_summary.json of each round. Load with pd.read_parquet("hf://arthrod/gliner-opf-ptbr-pii-v1/training_rounds.parquet").

Schema:

Example rows (first and last round):

[
 {
  "round": 1,
  "checkpoint": "e1_c1",
  "path": "checkpoint-e1_c1/",
  "train_dataset": "data/splits/oai_pf_train_natural_chunk1.jsonl",
  "num_train_examples": 304818,
  "num_train_windows": 326610,
  "optimizer_steps": 2552,
  "train_tokens": 27073808,
  "train_loss": 0.704006,
  "train_token_accuracy": 0.819779,
  "validation_loss": 0.421247,
  "validation_token_accuracy": 0.861019,
  "elapsed_s": 1310.870837
 },
 {
  "round": 9,
  "checkpoint": "e3_c3",
  "path": ".",
  "train_dataset": "data/splits/oai_pf_train_natural_chunk3.jsonl",
  "num_train_examples": 304816,
  "num_train_windows": 326238,
  "optimizer_steps": 2549,
  "train_tokens": 26925469,
  "train_loss": 0.133315,
  "train_token_accuracy": 0.956574,
  "validation_loss": 0.197111,
  "validation_token_accuracy": 0.930637,
  "elapsed_s": 1306.647574
 }
]

Per-entity F1 (e3_c3, span-typed, opf eval)

(Entries with zero gold in val are omitted.)

Comparison across the family

GLiNER holdout protocol (strict F1). This model was not evaluated with the GLiNER 11-source holdout protocol; for reference, the GLiNER models' released checkpoints:

Limitations

• All evaluation data is synthetic or from public PII datasets; there is no human-annotated real-world test set yet. Do not use it as the sole source of truth for redaction without human review.
• Weaker than mmBERT-small GLiNER on held-out sources (partial F1 0.712 vs 0.861 over four held-out sources) and on names, cities and middle names.
• location full address scores 0 on the val benchmark for every model.
• The label berco_de_tiradentes is in this model's label space but was not trained; use the mmBERT-small model for it.
• The training set (arthrod/oai-pf-ptbr-chunked-v2) is private, so the run cannot be reproduced from public data alone.
• Needs the opf package (installed from GitHub) and a GPU; it does not load with transformers or gliner.

License

Apache-2.0, same as the base model. Training/evaluation sources keep their own licenses (see their dataset cards).