Multi-Block Diffusion Language Models (MBD-LMs)

This repository contains the model weights for the paper Multi-Block Diffusion Language Models.

• Project Page: SJTU-DENG-Lab Project Page
• Repository: GitHub - SJTU-DENG-Lab/mbd-lms
• Paper: arXiv:2606.29215

Introduction

Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. Multi-Block Diffusion Language Models (MBD-LMs) extend this paradigm from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a running-set of consecutive blocks is decoded concurrently for inter-block parallelism.

This model is obtained by post-training BD-LMs with Multi-block Teacher Forcing (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded noise-groups conditioned on clean prefixes, with randomized noise-schedulers that better match MultiBD inference states.

For inference and evaluation setups, please refer to the optimized decoding algorithms in the Diffulex repository.

Citation

If you find this work useful, please consider citing:

@article{jin2026multiblock,
  title={Multi-Block Diffusion Language Models},
  author={Yijie Jin and Jiajun Xu and Yuxuan Liu and Chenkai Xu and Yi Tu and Jiajun Li and Dandan Tu and Xiaohui Yan and Kai Yu and Pengfei Liu and Zhijie Deng},
  journal={arXiv preprint arXiv:2606.29215},
  year={2026}
}