CamemBERTav2 — French Adverse-Event Mention Classifier

A binary classifier that detects whether a French text mentions an adverse event attributed to a vaccine or medicine. Fine-tuned from CamemBERTav2-base (DeBERTa-v2 architecture, Antoun et al. 2024) using a weak-supervision pipeline: LLM labels, targeted human audit, and self-training.

Designed as a reusable building block for French-language pharmacovigilance pipelines. The model, annotation guide, and evaluation protocol are released as commons for the clinical NLP community.

Intended uses

• Screening/triage of French health-related text for adverse-event mentions
• Pre-filter in NER pipelines (e.g., before entity extraction with GLiNER)
• Research on weak supervision, LLM-as-annotator, and low-resource health NLP

Not intended for: clinical decision-making, automated pharmacovigilance reporting, or use as a diagnostic tool. Predictions require human review.

Performance

Human gold standard (primary evaluation)

300 French tweets, double-coded by two independent annotators (95.0% agreement, Cohen's kappa = 0.900), adjudicated. Disjoint from all training data. Stratified by a pharmacovigilance lexicon: 176 lexicon matches, 124 non-matches. This set oversamples positives (151/149 split); corpus-wide metrics would differ.

Model-human agreement (94.0%–96.3%) bracketed human-human agreement (95.0%). This indicates comparable agreement, not superior performance.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "oliviercaron/camembertav2-adverse-event-fr"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

texts = [
    "Forte fièvre et douleurs musculaires 48h après ma 2e dose Pfizer",
    "Le vaccin est efficace à 95% contre les formes graves",
]

inputs = tokenizer(texts, truncation=True, padding=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)[:, 1].numpy()

for text, p in zip(texts, probs):
    label = "adverse event" if p >= 0.5 else "no adverse event"
    print(f"[{p:.3f}] {label}: {text[:80]}")

Training details

Data

LLM annotation

Four LLMs annotated a stratified 1,000-tweet sample (400 random, 300 vaccine-enriched, 300 symptom-enriched) from 50.2 million French COVID-19 tweets. Three converged (positive rates 7.8–10.1%, pairwise kappa 0.694–0.749); Gemini 2.0 Flash was excluded (23.2% positive rate). GPT-5.4 Nano was retained after auditing 25 positive labels. Total annotation cost: ~US$0.35.

Hyperparameters (Optuna, 26 trials)

Training: 21 minutes on one NVIDIA RTX 3070 (8 GB VRAM), FP16 mixed precision. Seed: 42.

Training code

The training/ folder contains the scripts used to build the dataset and train the model:

Note: Some scripts contain hardcoded paths that must be adapted to your environment. The corpus parquet file (50.2M tweets) is not distributed.

Limitations

1. Domain: trained on French COVID-19 tweets (2020–2021). Performance on other domains, time periods, or languages is untested.
2. Enriched evaluation: the gold set oversamples positives (~50/50); corpus-wide precision and recall would differ from the reported values. Only 1 of 124 out-of-lexicon tweets in the gold was positive.
3. Self-training circularity: 84% of training labels are pseudo-labels from the model's own preliminary version. Systematic blind spots may be amplified.
4. Not clinical text: trained and evaluated on public social-media text. Transfer to de-identified clinical notes remains untested.
5. Teacher audit scope: only 25 of ~78 GPT-5.4 Nano positive labels were audited; negative labels were not audited.

Ethical considerations

All tweets were public at collection time. Processing relies on GDPR Article 89 (research derogation). No individual is identified in the model output. The model detects mentions of adverse events as reported by social-media users; these are not confirmed medical events, diagnoses, or causal evidence.

Annotation guide

The frozen English annotation guide used for the gold standard is included in this repository as annotation_guide.md. It defines the task (ADR/AEFI detection), provides 11 decision rules, 14 reference examples, and describes the double-coding procedure.

Citation

If you use this model, please cite:

@inproceedings{caron2026weak,
  title     = {From {LLM} Weak Supervision to Human Gold:
               A Compact French Adverse-Event Mention Classifier},
  author    = {Caron, Olivier and Gimenez, Lucile and Benavent, Christophe},
  booktitle = {Building Commons for Clinical LLMs},
  year      = {2026},
  address   = {Paris, France}
}

Acknowledgments

The base model is CamemBERTav2 by Antoun et al. (2024).