UltraBERT-380M-clairic-extraction

A 380M-parameter extractor for Clairic-style requests: give it a text and a set of fields, each described by a short instruction, and it returns the matching spans for each field (one answer, a list, or nothing when the text has no answer). Fields can be questions ("What is the total due?") or entity types ("Every company mentioned in the text"), so one model covers extractive question answering and named-entity recognition.

UltraBERT-380M trained on the extraction config of tasksource-jev-typed-decisions (release v2026.10.24.1, 1M instruction/text pairs, 60k steps): extractive QA and span-tagging datasets from tasksource, rewritten as (instruction, text, spans) pairs. The model tags the answer tokens of the text (B/I/O) conditioned on the instruction. Inputs up to 2048 tokens.

It is the larger sibling of UltraBERT-150M-clairic-extraction, with the same interface: more accurate, especially on list fields, and better at leaving a field empty when the text has no answer, for about 2.5x the compute.

Not affiliated with Cloudflare; "Clairic-style" refers to the request format.

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

name = "tasksource/UltraBERT-380M-clairic-extraction"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name, trust_remote_code=True).eval()

def extract(text, fields):
    """fields: {name: instruction}. Returns {name: [extracted spans]}, [] when the text has no answer."""
    enc = tok(list(fields.values()), [text] * len(fields), truncation="only_second", max_length=2048,
              padding=True, return_offsets_mapping=True, return_tensors="pt")
    offsets = enc.pop("offset_mapping")
    with torch.no_grad():
        tags = model(**enc).logits.argmax(-1).tolist()  # 0 = O, 1 = B, 2 = I
    out = {}
    for f, field in enumerate(fields):
        spans, start, seq = [], None, enc.sequence_ids(f) + [None]
        for i, tag in enumerate(tags[f] + [0]):
            if start is not None and (seq[i] != 1 or tag != 2):
                spans.append(text[offsets[f][start][0]:offsets[f][i - 1][1]].strip().rstrip(",;:"))
                start = None
            if seq[i] == 1 and tag in (1, 2) and start is None:
                start = i
        out[field] = spans
    return out

text = "Invoice INV-2291 from Acme Corp to Globex, issued 3 March 2025. Total due: $1,240.50. Contact: billing@acme.com."
print(extract(text, {
    "invoice_number": "The invoice number.",
    "companies": "Every company mentioned in the text, as written.",
    "total": "The total amount due.",
    "phone": "The contact phone number.",
}))
# {'invoice_number': ['INV-2291'], 'companies': ['Acme Corp', 'Globex'], 'total': ['$1,240.50'], 'phone': []}

trust_remote_code=True is needed for the UltraBERT architecture.

Evaluation

Test split of the extraction config (revision v2026.10.20.7), 100 rows per source, procedurally generated sources excluded. text fields (one answer or none) are scored by SQuAD F1, the empty answer counting as an answer. list fields (every span of a kind) are scored by micro F1 over normalized values. Scores are means over sources.

The test sources are the training sources (their test splits), while the baselines see most of them zero-shot, so this measures performance on these tasks rather than on unseen ones.

Out of domain (MRQA test sets the model never trained on, plus FinEntity), F1:

GLiNER-multitask stays ahead out of domain.

Training data

The extraction config gathers extractive QA (SQuAD 2.0, Natural Questions, HotpotQA, NewsQA, DuoRC, QuAC, CUAD, Qasper, TimeQA, WebSRC, SWDE, and others) and span tagging (CoNLL-2003, OntoNotes, Few-NERD, WNUT, CrossNER, biomedical, legal and financial NER, skills, aspects, keyphrases, and others), plus procedurally generated documents. A few sources (e.g. GUM, PreCo) are licensed for non-commercial use only; check the dataset card for per-source licenses.

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    year = "2024",
    address = "Torino, Italia",
    pages = "15655--15684",
    url = "https://aclanthology.org/2024.lrec-main.1361",
}