Reducio-VAE Model Card

This model is a 3D VAE that encodes video into a compact latent space conditioned on a content frame. It compresses a video by a factor of \(\frac{T}{4}\times\frac{H}{32}\times\frac{W}{32}\), enabling 4096x downsampling. It is part of the Reducio-DiT, which is a video generation method. Codebase available here.

Model Details

Model Sources

• Repository: GitHub Repository
• Paper: arXiv

Uses

Common use scenario is described here.

Direct Use

The model is typically used for supporting training a video diffusion model. After using this model to convert the data to the latent space, you can train your own diffusion model on the extremely compressed latent space.

Results

Results

Metrics on 1K Pexels validation set and UCF-101:

Citation

BibTeX:

@article{tian2024reducio,
      title={REDUCIO! Generating 1024*1024 Video within 16 Seconds using Extremely Compressed Motion Latents}, 
      author={Tian, Rui and Dai, Qi and Bao, Jianmin and Qiu, Kai and Yang, Yifan and Luo, Chong and Wu, Zuxuan and Jiang, Yu-Gang},
      journal={arXiv preprint arXiv:2411.13552},
      year={2024}
}