ConfliBERT-2 (base)

A long-context encoder for political conflict research: ModernBERT-base with continued masked-language-model pretraining on 22 billion tokens of conflict, diplomacy and political text, through 2026.

ConfliBERT (2021) gave conflict researchers a domain-pretrained encoder, but it inherits BERT's 512-token window, a 2019 architecture, and training text that ends in 2020. ConfliBERT-2 keeps the idea and replaces the encoder: ModernBERT's 8,192-token context and speed, plus one full pass of domain pretraining on a corpus that runs from Cold War diplomatic records to 2026 web text. It is meant to be fine-tuned: a few hundred labeled examples teach it your codebook, and it then codes millions of documents on one GPU, with no data leaving your machine.

A large version (395M parameters, from ModernBERT-large) is training and will be released as eventdata-utd/ConfliBERT-2-large.

What it is for

• Event coding at scale. Relevance filtering, event-type classification, actor and location tagging (NER) over news archives and monitoring reports.
• Long documents. Full hearings, country reports, peace agreements, Security Council records: the model reads up to 8,192 tokens (about twelve pages) in one pass, where the 2021 model stops at one page. In the IndiaPoliceEvents corpus, when the evidence sits past token 512 the truncated 2021 model's recall falls from .81 to .23.
• Judgment labels. Codebook categories that depend on the whole document (a monthly crisis trend, a hearing's purpose, a severity grade). Fine-tuned encoders reach .79 recall on CrisisWatch deteriorations where a zero-shot frontier LLM has precision .12.
• Local, cheap, reproducible. About 0.7 GPU-hours per million documents at 512 tokens; sensitive sources never leave the machine.

English only. Not a chat model: it reads text and returns labels or fills blanks; it does not generate text.

How to use

Fill-mask (what pretraining teaches):

from transformers import pipeline
fill = pipeline("fill-mask", model="eventdata-utd/ConfliBERT-2-base")
fill("Fighting between the army and the Rapid Support Forces displaced millions in [MASK].")

Fine-tune for classification (the intended use):

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("eventdata-utd/ConfliBERT-2-base")
model = AutoModelForSequenceClassification.from_pretrained(
    "eventdata-utd/ConfliBERT-2-base", num_labels=NUM_LABELS)
# standard Trainer fine-tuning; inputs up to 8,192 tokens.

Token classification (NER) works the same way with AutoModelForTokenClassification. Notes:

• Install flash-attn for the fastest long-input inference; without it the model runs on PyTorch SDPA with the same outputs.
• The tokenizer is ModernBERT's (cased, 50,368 tokens). ConfliBERT-2 is not a drop-in replacement for the 2021 model: fine-tuned heads and tokenized datasets have to be rebuilt.
• For NER with pre-split words, pass add_prefix_space=True to the tokenizer; a missing prefix space costs F1 on this tokenizer family.
• Learning rates that worked in our benchmark: 2e-5 to 8e-5 for classification, 8e-5 to 2.4e-4 for NER, selected on a validation split.

Evaluation

All models fine-tuned under one protocol: learning rate chosen on the validation split (seed 123), three seeds at that rate, test F1 reported as the mean. NER tasks use the prefix-space fix and the extended learning-rate grid.

• ConfliBERT-2 base beats ModernBERT-base on six of nine tasks and matches ModernBERT-large, which has 2.6 times the parameters.
• The 2021 model keeps a 0.9-point lead on the mean, most of it from one task (IndiaPoliceEvents documents, 6 points). The clearest gain over both ModernBERT sizes is CAMEO NER, +2.5 over ModernBERT-base.
• Pseudo-perplexity on 2,000 held-out blocks of the training corpus (fixed 30% mask, 613,809 masked tokens): ModernBERT-base 4.446, ModernBERT-large 3.368, ConfliBERT-2 base 3.419.
• Harness note: the ModernBERT-base and ConfliBERT-2021 rows come from the pilot harness (RTX 4090, transformers 4.x); ConfliBERT-2 and ModernBERT-large ran on NCSA Delta (A100, transformers 5.17). A same-harness ModernBERT-base rerun is in progress and will replace that column.

Task sources: the ConfliBERT benchmark (SATP, InsightCrime, CAMEO, re3d, 20 Newsgroups, BBC) and IndiaPoliceEvents (Halterman et al., 2021).

Training

The released weights carry a dense MLM head and no pinned attention backend, like stock ModernBERT.

Corpus

Sixteen sources, every one used in full, exact duplicates removed (65,758 documents, 0.26%). Held out and never trained on: a 0.3% random sample of every source and all documents dated 2021.

Filtered web text is about four fifths of the tokens, the curated sources one fifth. Dates run from the 1940s (FRUS, Hansard) to 2026.

Limitations

• English only; the corpus over-represents English-language news and the US, UK and UN institutional record.
• News-derived text carries the coverage and reporting biases of its sources; performance varies by event type and region.
• Pretraining knowledge of post-2020 events is broad, not authoritative: fill-mask outputs are not facts.
• A fine-tuned classifier is a measurement instrument. Report the model, the context window, the inference mode, the tuning protocol and the seeds; each moved our numbers.
• Not a substitute for human review in sensitive applications.

Code and reproduction

Everything needed to rebuild the corpus, pack it, train, and run the evaluation harness is in github.com/shreyasmeher/conflibert-v2, including the Slurm jobs used on Delta. The internal run name of these weights is conflibert-v2-cb2-wsd-full.

Citation

@misc{meher2026conflibert2,
  author       = {Meher, Shreyas and Brandt, Patrick T. and Chakma, Aunabil and D'Orazio, Vito and Medapati, Yamini and Osorio, Javier},
  title        = {ConfliBERT-2: A Long-Context Encoder for Political Conflict Research},
  year         = {2026},
  howpublished = {Hugging Face model repository, \url{https://huggingface.co/eventdata-utd/ConfliBERT-2-base}}
}

Please also cite ModernBERT (Warner et al., 2024) and ConfliBERT (Hu et al., 2022, NAACL).

Acknowledgements

Supported by NSF award 2311142. This work used Delta at NCSA / University of Illinois through allocation CIS220162 from the ACCESS program (NSF grants 2138259, 2138286, 2138307, 2137603, 2138296). Corpus contributions from the ConfliBERT team at UT Dallas, the University of Arizona, and West Virginia University.