🎨 LightVAE

⚡ Efficient Video Autoencoder (VAE) Model Collection

From Official Models to Lightx2v Distilled Optimized Versions - Balancing Quality, Speed and Memory

---

---

For VAE, the LightX2V team has conducted a series of deep optimizations, deriving two major series: LightVAE and LightTAE, which significantly reduce memory consumption and improve inference speed while maintaining high quality.

💡 Core Advantages

---

📦 Available Models

🎯 Wan2.1 Series VAE

🎯 Wan2.2 Series VAE

---

📊 Wan2.1 Series Performance Comparison

• Precision: BF16
• Test Hardware: NVIDIA H100

Video Reconstruction (5s 81-frame video)

Video Generation

Task: s2v(speech to video)
Model: seko-talk

📊 Wan2.2 Series Performance Comparison

• Precision: BF16
• Test Hardware: NVIDIA H100

Video Reconstruction

Video Generation

Task: t2v(text to video)
Model: Wan2.2-TI2V-5B

🎯 Model Selection Recommendations

Selection by Use Case

🔥 Our Optimization Results Comparison

📑 Todo List

• LightX2V integration
• ComfyUI integration
• Training & Distillation Code

🚀 Usage

Download VAE Models

# Download Wan2.1 official VAE
huggingface-cli download lightx2v/Autoencoders \
    --local-dir ./models/vae/

🧪 Video Reconstruction Test

We provide a standalone script vid_recon.py to test VAE models independently. This script reads a video, encodes it through VAE, then decodes it back to verify the reconstruction quality.

Script Location: LightX2V/lightx2v/models/video_encoders/hf/vid_recon.py

git clone https://github.com/ModelTC/LightX2V.git
cd LightX2V

1. Test Official VAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/Wan2.1_VAE.pth \
    --model_type vaew2_1 \
    --device cuda \
    --dtype bfloat16

2. Test Official VAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/Wan2.2_VAE.pth \
    --model_type vaew2_2 \
    --device cuda \
    --dtype bfloat16

3. Test LightTAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lighttaew2_1.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

4. Test LightTAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lighttaew2_2.pth \
    --model_type taew2_2 \
    --device cuda \
    --dtype bfloat16

5. Test LightVAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lightvaew2_1.pth \
    --model_type vaew2_1 \
    --device cuda \
    --dtype bfloat16 \
    --use_lightvae

6. Test TAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/taew2_1.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

7. Test TAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/taew2_2.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

Use in LightX2V

Specify the VAE path in the configuration file:

Using Official VAE Series:

{

    "vae_path": "./models/vae/Wan2.1_VAE.pth"
}

Using LightVAE Series:

{
    "use_lightvae": true,
    "vae_path": "./models/vae/lightvaew2_1.pth"
}

Using LightTAE Series:

{
    "use_tae": true,
    "need_scaled": true,
    "tae_path": "./models/vae/lighttaew2_1.pth"
}

Using TAE Series:

{
    "use_tae": true,
    "tae_path": "./models/vae/taew2_1.pth"
}

Then run the inference script:

cd LightX2V/scripts
bash wan/run_wan_i2v.sh  # or other inference scripts

Use in ComfyUI

please refer to https://github.com/ModelTC/ComfyUI-LightVAE

⚠️ Important Notes

1. Compatibility

• Wan2.1 series VAE only works with Wan2.1 backbone models
• Wan2.2 series VAE only works with Wan2.2 backbone models
• Do not mix different versions of VAE and backbone models

📚 Related Resources

Documentation Links

• LightX2V Quick Start: Quick Start Documentation
• Model Structure Description: Model Structure Documentation
• taeHV Project: GitHub - madebyollin/taeHV

Related Models

• Wan2.1 Backbone Models: Wan-AI Model Collection
• Wan2.2 Backbone Models: Wan-AI/Wan2.2-TI2V-5B
• LightX2V Optimized Models: lightx2v Model Collection

---

🤝 Community & Support

• GitHub Issues: https://github.com/ModelTC/LightX2V/issues
• HuggingFace: https://huggingface.co/lightx2v
• LightX2V Homepage: https://github.com/ModelTC/LightX2V

If you find this project helpful, please give us a ⭐ on GitHub