ConfliBERT-2 (large)

A long-context encoder for political conflict research: ModernBERT-large with continued masked-language-model pretraining on 22 billion tokens of conflict, diplomacy and political text, through 2026.

ConfliBERT (2021) gave conflict researchers a domain-pretrained encoder, but it inherits BERT's 512-token window, a 2019 architecture, and training text that ends in 2020. ConfliBERT-2 keeps the idea and replaces the encoder: ModernBERT's 8,192-token context and speed, plus one full pass of domain pretraining on a corpus that runs from Cold War diplomatic records to 2026 web text. It is meant to be fine-tuned: a few hundred labeled examples teach it your codebook, and it then codes millions of documents on one GPU, with no data leaving your machine.

This is the large version, with 395M parameters. A smaller, faster base version (150M) is at eventdata-utd/ConfliBERT-2-base.

What it is for

• Event coding at scale. Relevance filtering, event-type classification, actor and location tagging (NER) over news archives and monitoring reports.
• Long documents. Full hearings, country reports, peace agreements, Security Council records: the model reads up to 8,192 tokens (about twelve pages) in one pass, where the 2021 model stops at one page. In the IndiaPoliceEvents corpus, when the evidence sits past token 512 the truncated 2021 model's recall falls from .81 to .23.
• Judgment labels. Codebook categories that depend on the whole document (a monthly crisis trend, a hearing's purpose, a severity grade). Fine-tuned encoders reach .79 recall on CrisisWatch deteriorations where a zero-shot frontier LLM has precision .12.
• Local, cheap, reproducible. About [[TBD-GPUH-PER-M]] GPU-hours per million documents at 512 tokens; sensitive sources never leave the machine. The base version is about 0.7 GPU-hours per million documents, if throughput matters more than the last point of accuracy.

English only. Not a chat model: it reads text and returns labels or fills blanks; it does not generate text.

How to use

Fill-mask (what pretraining teaches):

from transformers import pipeline
fill = pipeline("fill-mask", model="eventdata-utd/ConfliBERT-2-large")
fill("Fighting between the army and the Rapid Support Forces displaced millions in [MASK].")

Fine-tune for classification (the intended use):

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("eventdata-utd/ConfliBERT-2-large")
model = AutoModelForSequenceClassification.from_pretrained(
    "eventdata-utd/ConfliBERT-2-large", num_labels=NUM_LABELS)
# standard Trainer fine-tuning; inputs up to 8,192 tokens.

Token classification (NER) works the same way with AutoModelForTokenClassification. Notes:

• Install flash-attn for the fastest long-input inference; without it the model runs on PyTorch SDPA with the same outputs.
• The tokenizer is ModernBERT's (cased, 50,368 tokens). ConfliBERT-2 is not a drop-in replacement for the 2021 model: fine-tuned heads and tokenized datasets have to be rebuilt.
• For NER with pre-split words, pass add_prefix_space=True to the tokenizer; a missing prefix space costs F1 on this tokenizer family.
• Learning rates that worked in our benchmark: [[TBD-LR-CLS]] for classification, [[TBD-LR-NER]] for NER, selected on a validation split.

Evaluation

All models fine-tuned under one protocol: learning rate chosen on the validation split (seed 123), three seeds at that rate, test F1 reported as the mean. NER tasks use the prefix-space fix and the extended learning-rate grid.

Bold the best score in each row once the large column is in.

• [[TBD: ConfliBERT-2 large vs ModernBERT-large, the like-for-like comparison: tasks won out of nine, and the mean difference.]]
• [[TBD: ConfliBERT-2 large vs ConfliBERT (2021): whether the 0.9-point gap on the mean closes, and how IndiaPoliceEvents documents moves.]]
• Pseudo-perplexity on 2,000 held-out blocks of the training corpus (fixed 30% mask, 613,809 masked tokens): ModernBERT-base 4.446, ModernBERT-large 3.368, ConfliBERT-2 base 3.419, ConfliBERT-2 large [[TBD-PPL]].
• Harness note: the ModernBERT-base and ConfliBERT-2021 rows come from the pilot harness (RTX 4090, transformers 4.x); ModernBERT-large and both ConfliBERT-2 models ran on NCSA Delta (A100, transformers 5.17). [[TBD: replace the ModernBERT-base column with the Delta rerun (eval ModernBERT-base-delta) and drop this sentence for that column.]]

Task sources: the ConfliBERT benchmark (SATP, InsightCrime, CAMEO, re3d, 20 Newsgroups, BBC) and IndiaPoliceEvents (Halterman et al., 2021).

Training

The released weights carry a dense MLM head and no pinned attention backend, like stock ModernBERT.

Corpus

The same corpus as the base version. Sixteen sources, every one used in full, exact duplicates removed (65,758 documents, 0.26%). Held out and never trained on: a 0.3% random sample of every source and all documents dated 2021.

Filtered web text is about four fifths of the tokens, the curated sources one fifth. Dates run from the 1940s (FRUS, Hansard) to 2026.

Limitations

• English only; the corpus over-represents English-language news and the US, UK and UN institutional record.
• News-derived text carries the coverage and reporting biases of its sources; performance varies by event type and region.
• Pretraining knowledge of post-2020 events is broad, not authoritative: fill-mask outputs are not facts.
• A fine-tuned classifier is a measurement instrument. Report the model, the context window, the inference mode, the tuning protocol and the seeds; each moved our numbers.
• Not a substitute for human review in sensitive applications.

Code and reproduction

Everything needed to rebuild the corpus, pack it, train, and run the evaluation harness is in github.com/shreyasmeher/conflibert-v2, including the Slurm jobs used on Delta. The internal run name of these weights is conflibert-v2-large-cb2-wsd-full.

Citation

@misc{meher2026conflibert2,
  author       = {Meher, Shreyas and Brandt, Patrick T. and Chakma, Aunabil and D'Orazio, Vito and Medapati, Yamini and Osorio, Javier},
  title        = {ConfliBERT-2: A Long-Context Encoder for Political Conflict Research},
  year         = {2026},
  howpublished = {Hugging Face model repository, \url{https://huggingface.co/eventdata-utd/ConfliBERT-2-large}}
}

Please also cite ModernBERT (Warner et al., 2024) and ConfliBERT (Hu et al., 2022, NAACL).

Acknowledgements

Supported by NSF award 2311142. This work used Delta at NCSA / University of Illinois through allocation CIS220162 from the ACCESS program (NSF grants 2138259, 2138286, 2138307, 2137603, 2138296). Corpus contributions from the ConfliBERT team at UT Dallas, the University of Arizona, and West Virginia University.