Agnes-Video-2.0-MoE

Agnes-Video-2.0-MoE is an audio-video generation model based on DaVinci-MagiHuman. It accepts text, video, and audio inputs and generates video and audio. With TIRx kernels, a 5-second 480×272 clip at 32 denoising steps now takes 13.641 s on an NVIDIA B200.

Model Description

Agnes-Video-2.0-MoE uses a 40-layer, single-stream multimodal DiT. The shared feed-forward networks in layers 4–35 are converted to three-expert SparseMoE blocks with Top-1 routing. checkpoint_parts/ is the EENP p60 checkpoint: modality-aware neuron pruning across video, audio, and text, with elastic Quality and Fast widths.

The uploaded checkpoint uses two nested widths. Quality retains 65% of intermediate neurons, and Fast retains 40% as a high-score subset of Quality. A 32-step run uses 8 Quality steps and 24 Fast steps. These percentages describe routed FFN neuron widths, not the fraction of total model parameters removed. The parameter counts above are the unpruned SparseMoE model. Separately obtained text encoder and VAE weights are not included.

Video Demos

The examples below are text-conditioned generations. Click a preview to open its MP4 video.

Results

Open-Generator Comparison

The comparison uses 32 Koala clips. The “Ours” row is the unpruned SparseMoE checkpoint (28.7B stored / 15.3B active parameters). The weights in checkpoint_parts/ are the EENP p60 checkpoint.

Stage Results and Quality–Speed Trade-off

checkpoint_parts/ is this EENP 60% checkpoint. Relative to unpruned SparseMoE, it reduces DiT denoising latency from 26.83 s to 20.92 s (22.0% lower; 1.28× speedup). VBench 2.0 changes from 0.519 to 0.479. On the paired diagnostic set, FVD changes from 64.267 to 121.012 and CLIP-Sim from 0.9186 to 0.8227, reflecting the quality–speed trade-off.

Inference

Set the component paths in the inference configuration, then run:

export OMNIVAE_CODE_ROOT=/path/to/OmniVAE/vae
export OMNIVAE_RELEASE_ROOT=/path/to/OmniVAE/release

cat checkpoint_parts/ckpt_best.pt.part-* > ckpt_eenp_p60.pt

TIRX_ELASTIC=1 bash inference_code/infer.sh \
  --config-load-path /path/to/config_moe32_omnivae_init.json \
  --ckpt_dir ckpt_eenp_p60.pt \
  --generate \
  --prompt "A small sailboat crossing a calm lake at sunset." \
  --seconds 5 \
  --num_inference_steps 32 \
  --cfg_number 2 \
  --save_path_prefix outputs/demo

TIRx kernels

checkpoint_parts/ is the EENP p60 checkpoint, and denoising uses TIRx kernels. On an NVIDIA B200, one 5-second 480×272 clip at 32 steps now takes 13.641 s (denoising only). That is 4.150 s faster than the 17.791 s PyTorch path, a 23.33% reduction.

TIRX_ELASTIC=1 is the path for these weights. It needs tirx_kernels on Blackwell (sm_100).

Model Components

License and Third-Party Dependencies

Agnes-Video-2.0-MoE checkpoints contain only Agnes-Video-2.0-MoE DiT/MoE weights. No pretrained base-model weights or third-party model weights are included or redistributed.

Agnes-Video-2.0-MoE code and model weights are provided under the Apache License 2.0, subject to applicable upstream license requirements. See LICENSE and NOTICE.

Agnes-Video-2.0-MoE is developed based on DaVinci-MagiHuman. We acknowledge the original project and retain applicable license notices and attribution.

For inference, users must separately obtain required base-model and auxiliary-model weights from their respective official repositories.

Google T5Gemma is an external text encoder governed by the Gemma Terms of Use, not Apache 2.0. Users must obtain T5Gemma independently and comply with its applicable terms.

Third-party models and dependencies remain subject to their respective licenses. The Apache License 2.0 for Agnes-Video-2.0-MoE does not extend to these external components.

References:

• DaVinci-MagiHuman
• T5Gemma
• Gemma Terms of Use