ModernCamemBERT-bio-v2-base
ModernCamemBERT-bio-v2-base is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-base with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 70.0 micro-F1 at 150M parameters. A larger version is available as ModernCamemBERT-bio-v2-large.
Authors
Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris
Overview
ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Instead of plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The aim is to inject structured coding knowledge without adding a task-specific head. This is the base-size document specialist.
| Architecture | ModernBERT |
|---|---|
| Parameters | 150M |
| Context length | 8,192 tokens |
| Language | French |
| Base model | almanach/ModernCamemBERT-bio-base |
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "rntc/ModernCamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))
Evaluation
Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.
| Model | FrACCO-30 | FrACCO-100 | CANTEMIST | MORFITT | MedDialog-FR | Avg |
|---|---|---|---|---|---|---|
| ModernCamemBERT-bio-v2-large | 82.0 | 67.8 | 74.6 | 74.3 | 65.2 | 72.8 |
| ModernCamemBERT-bio-v2-base | 78.0 | 63.7 | 70.9 | 73.5 | 63.7 | 70.0 |
| ModernCamemBERT-bio-base | 74.2 | 61.1 | 71.2 | 72.9 | 63.7 | 68.6 |
| DoctoModernBERT | 74.0 | 58.3 | 69.3 | 73.9 | 64.7 | 68.0 |
| DrBERT | 54.2 | 39.6 | 41.2 | 71.0 | 64.1 | 54.0 |
| DoctoBERT | 51.6 | 34.7 | 36.2 | 73.5 | 62.1 | 51.6 |
| CamemBERT-bio | 44.0 | 22.3 | 17.7 | 70.9 | 45.8 | 40.1 |
At base size, ModernCamemBERT-bio-v2-base leads the average and scores 1.4 points above its ModernCamemBERT-bio-base starting point. ModernCamemBERT-bio-v2-large scores 2.8 points higher at 350M parameters.
Environmental impact
The OntoBook phase reported here ran for about 1.3 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.04 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.
License
MIT
Citation
@inproceedings{touchent:hal-05697506,
TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
URL = {https://hal.science/hal-05697506},
BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
ADDRESS = {Palma de Mallorca, Spain},
YEAR = {2026},
MONTH = May,
PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
HAL_ID = {hal-05697506},
HAL_VERSION = {v1},
}
@misc{touchent2026causallanguagemodelingdetour,
title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
author={Rian Touchent and Eric de la Clergerie},
year={2026},
eprint={2605.12438},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.12438},
}