ByteDance/Bernini-Diffusers-v2

Latent Semantic Planning for Video Diffusion

Chenchen Liu^*, Junyi Chen^*, Lei Li^*, Lu Chi^*,§, Mingzhen Sun^*, Zhuoying Li^*, Yi Fu, Ruoyu Guo, Yiheng Wu, Ge Bai, Zehuan Yuan^✉

^* Equal contribution  ^✉ Corresponding author  ^§ Project lead

🎉 News

• [2026-06-11] We open-sourced the inference code and model weights of the full Bernini (Bernini).
• [2026-05-22] We released our paper Bernini: Latent Semantic Planning for Video Diffusion.

✨ Highlights

Bernini is a unified framework for video generation and editing that combines an MLLM-based semantic planner with a DiT-based renderer.

Bernini-Diffusers-v2 packages the full semantic-planning pipeline: a Qwen2.5-VL planner, Bernini planning weights, and Wan2.2 diffusion components in one self-contained diffusers-format directory.

Compared with the renderer-only Bernini-R releases, Bernini-Diffusers-v2 is recommended when you need stronger instruction following, multi-step semantic planning, and better handling of complex video generation or editing requests. Compared with the first Bernini-Diffusers release, v2 uses a training recipe that warms up the connector for thousands of steps before co-training, improving reference-guided video editing and OpenS2V performance.

🧾 Model card

Benchmark snapshot

On video editing, Bernini reaches the first tier among leading closed-source commercial models in our internal arena evaluation based on blind human pairwise comparisons.

📦 Package layout

This release is a self-contained diffusers-format directory. Pass the downloaded Bernini-Diffusers-v2 directory directly to --config.

Bernini-Diffusers-v2/
  bernini/
  mllm/
  scheduler/
  t5_text_encoder/
  t5_tokenizer/
  vae/
  config.json
  transformer_config.json
  transformer_2_config.json

At runtime:

• bernini/ provides the Bernini planning checkpoint.
• mllm/ provides the Qwen2.5-VL planner assets.
• transformer_config.json and transformer_2_config.json define the Wan2.2 diffusion decoder components used by the full pipeline.
• t5_text_encoder/, t5_tokenizer/, vae/, and scheduler/ provide the base diffusion modules required for inference.

📥 Download

pip install -U "huggingface_hub"
hf download ByteDance/Bernini-Diffusers-v2 \
    --local-dir pretrained_models/Bernini-Diffusers-v2

🚀 Usage

The official inference code is available in the Bernini repository.

Installation

git clone https://github.com/bytedance/Bernini.git bernini && cd bernini
pip install -r requirements.txt
# Open-VeOmni is required. Install it with --no-deps so it does not pull in a
# different torch build and override the pinned torch==2.7.1+cu126:
pip install --no-deps git+https://github.com/ByteDance-Seed/VeOmni.git@v0.1.11

Recommended environment:

• Python 3.11.2
• PyTorch 2.7.1+cu126
• CUDA toolkit 12.6
• GPU Hopper GPUs (H100/H800/H200) are recommended for best performance

Load the model

Pass the downloaded directory directly as --config:

python infer_single_gpu.py --config pretrained_models/Bernini-Diffusers-v2 \
    --case assets/testcases/i2i/i2i.json --num_frames 1

Prompt enhancer (highly recommended)

--use_pe enhances the prompt through an OpenAI-compatible endpoint and is recommended for best generation quality.

export BERNINI_PE_API_KEY=...      # or OPENAI_API_KEY
export BERNINI_PE_BASE_URL=...     # or OPENAI_BASE_URL
export BERNINI_PE_MODEL=...        # vision-capable chat model

Gradio demo

# Single GPU
python gradio_demo.py --config pretrained_models/Bernini-Diffusers-v2 --port 7860

# 8 GPUs, 8-way Ulysses sequence parallel
torchrun --nproc-per-node 8 gradio_demo.py --ulysses 8 \
    --config pretrained_models/Bernini-Diffusers-v2 \
    --port 7860 --share

Run scripts

The scripts/bernini_v2/ directory in the Bernini repo provides ready-to-run task launchers for Bernini-Diffusers-v2:

• run_t2i.sh
• run_i2i.sh
• run_t2v.sh
• run_v2v.sh
• run_rv2v.sh
• run_r2v.sh
• run_gradio.sh

You can override the model directory with:

export BERNINI_CONFIG=/path/to/Bernini-Diffusers-v2

📑 Citation

If you use Bernini in your research, please cite:

@article{bernini,
  title   = {Bernini: Latent Semantic Planning for Video Diffusion},
  author  = {Chenchen Liu and Junyi Chen and Lei Li and Lu Chi and Mingzhen Sun and Zhuoying Li and Yi Fu and Ruoyu Guo and Yiheng Wu and Ge Bai and Zehuan Yuan},
  journal = {arXiv preprint arXiv:2605.22344},
  year    = {2026}
}

🙏 Acknowledgements

Bernini builds on several outstanding open-source projects:

• Wan2.2-T2V-A14B
• Qwen2.5-VL-7B-Instruct
• VeOmni

📄 License

Apache License 2.0.