DoctoModernBERT-fr-base-diffusion-v0.1

A text diffusion model based on DoctoModernBERT-fr-base, finetuned on a medical instruction-following dataset.

Unlike an autoregressive LLM, which decodes strictly left to right one token at a time, this model works by iterative denoising. It starts from a fully masked canvas and, at each step, predicts all masked positions at once and keeps only the most confident ones. It still moves through the sequence one block at a time, so generation is parallel inside each block but left to right across blocks.

This is an exploratory checkpoint. It's small and lightly trained, so expect mistakes and weaker answers.

🚀 How to Use

Training and generation are done through the dLLM library.

Setup

git clone https://github.com/ZHZisZZ/dllm
cd dllm
pip install -e .

Sampling

import dllm

model_id = "bofenghuang/doctomodernbert-fr-base-diffusion-v0.1"  # or a local path

model = dllm.utils.get_model(model_name_or_path=model_id).eval()
tokenizer = dllm.utils.get_tokenizer(model_name_or_path=model_id)
sampler = dllm.core.samplers.MDLMSampler(model=model, tokenizer=tokenizer)

config = dllm.core.samplers.MDLMSamplerConfig(
    steps=128,             # total diffusion steps
    max_new_tokens=128,    # length of the generated answer span
    block_size=32,         # semi-autoregressive block size
    temperature=0.0,       # 0.0 = greedy / low-confidence remasking
    remasking="low_confidence",
)

messages = [
    [{"role": "user", "content": "Quels sont les symptômes typiques d'une infection urinaire ?"}],
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True)
outputs = sampler.sample(inputs, config, return_dict=True)
print(dllm.utils.sample_trim(tokenizer, outputs.sequences.tolist(), inputs)[0])

Interactive chat

python -u examples/bert/chat.py --model_name_or_path "bofenghuang/doctomodernbert-fr-base-diffusion-v0.1"

The chat template wraps turns as [SYS] … [/SYS], [Question] … [/Question], [Answer] … [/Answer].