CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

📄 Paper | 💻 Code | 🤗 Model

---

Overview

CLEAR is a mask-free video subtitle removal framework built on top of Wan2.1. It achieves end-to-end inference through a two-stage context-aware adaptive learning design — requiring no external text detection or segmentation modules at inference time.

• 🎯 Mask-Free: No OCR, no segmentation — just input a video and get clean output
• 🚀 Parameter Efficient: Only 0.77% trainable parameters via LoRA (rank=64)
• 🌍 Zero-Shot Cross-Lingual: Trained on Chinese subtitles, generalizes to English, Japanese, Korean, French, German, Russian, and Arabic
• 📈 State-of-the-Art: +6.77 dB PSNR and −74.7% VFID over best mask-dependent baselines

---

Performance

Evaluated on Chinese subtitle test set (rank=64, steps=5, cfg=1.0, lora_scale=1.0):

---

Usage

Requirements

# 1. Install Wan2.1
git clone https://github.com/Wan-Video/Wan2.1.git && cd Wan2.1
pip install -r requirements.txt

# 2. Install DiffSynth-Studio
git clone https://github.com/modelscope/DiffSynth-Studio.git && cd DiffSynth-Studio
pip install -e .

# 3. Download Wan2.1 base weights
huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir ./models/Wan2.1-T2V-1.3B

# 4. Install CLEAR dependencies
pip install -r requirements.txt

Inference (Mask-Free)

python inference.py \
    --model_base_path /path/to/Wan2.1-T2V-1.3B \
    --lora_checkpoint ./checkpoints/CLEAR-mask-free-subtitle-removal.pt \
    --lora_rank 64 \
    --lora_scale 1.0 \
    --input_video input_video.mp4 \
    --output_dir ./results \
    --num_steps 5 \
    --cfg_scale 1.0 \
    --use_sliding_window \
    --create_comparison

Key Inference Parameters

---

Method

CLEAR uses a two-stage training pipeline with mask-free inference:

Stage I — Self-Supervised Prior Learning Dual ResNet-50 encoders extract disentangled occlusion features from paired (clean, subtitled) video frames using orthogonality constraints and focal-weighted pseudo-labels — no manual annotation needed.

Stage II — Adaptive Weighting Learning A lightweight occlusion head (~2.1M params) and LoRA adaptation are jointly trained on the frozen Wan2.1 backbone. Dynamic alpha scheduling prevents over-reliance on noisy Stage I priors.

Inference — End-to-End, Mask-Free The Stage I encoder is not needed at inference time. Adaptive weighting is fully internalized into the LoRA-augmented attention, enabling single-pass DDIM generation from a subtitled video alone.

---

Ablation

Contact

• Qingdong He — heqingdong@alu.uestc.edu.cn
• Chaoyi Wang — chaoyiwang@mail.sim.ac.cn

---

Acknowledgements

Built upon Wan2.1, DiffSynth-Studio, and PEFT.

Disclaimer

This model is released for research purposes only. We explicitly prohibit use for generating misinformation or any malicious application. Please ensure compliance with applicable laws and the upstream Wan2.1 Community License.