Kandinsky S2V 1.0 Pro

Speech-to-video generation: an avatar video is produced from a reference frame and a speech track. The whole figure is animated, not only the head. The audio is a condition — the model does not generate sound, it animates the person in the reference frame to match the speech it is given.

Examples

Each is one reference frame plus one speech track. More examples — other languages, cartoon characters and animals, long videos — are on the project page.

Pipeline

Latent diffusion with flow matching.

Qwen2.5-Omni-7B supplies both conditions: Thinker hidden states for the text prompt, and audio-tower features (3584-dim, 25 tokens/s) for the speech.
CLIP ViT-L/14 supplies a pooled text embedding.
HunyuanVideo 3D VAE encodes and decodes video.
DiT (21.4B parameters, 60 visual blocks) is the generative backbone. Speech enters through per-layer cross-attention adapters on 18 of the visual blocks, frame-aligned so each latent frame attends only to its own audio tokens. FramePack memory at three temporal scales carries context from one chunk to the next.

Kandinsky-S2V-1.0-Pro-distill — pi-Flow distillation of this checkpoint, 16 steps instead of 50.

File	Size	Precision
`model/kandinsky_s2v_pro.safetensors`	39.8 GiB	bf16, with the time embedding and all normalisations in fp32

Text encoders and the VAE are not part of this repository; the code fetches them from their own repositories on first run:

Component	Source
Qwen2.5-Omni-7B	`Qwen/Qwen2.5-Omni-7B`
CLIP ViT-L/14	`openai/clip-vit-large-patch14`
HunyuanVideo 3D VAE	`hunyuanvideo-community/HunyuanVideo` (`vae/`)

Output

25 fps in SD or, with the *_hd.yaml configurations, HD resolution, at the reference frame's aspect ratio. Long speech is generated in 5-second chunks.

Usage

The code is in kandinskylab/kandinsky-s2v:

git clone https://github.com/kandinskylab/kandinsky-s2v
cd kandinsky-s2v
uv sync
uv run kandy-s2v generate \
    --config src/kandinsky_s2v/configs/s2v_pro.yaml \
    --audio speech.wav \
    --image reference.png \
    --out out.mp4

The first run downloads this checkpoint, the encoders and the VAE from the Hub. For HD, pass src/kandinsky_s2v/configs/s2v_pro_hd.yaml instead; it uses the same weights. To speed a generation up, split it across the GPUs of one node — see the repository README.

Performance

Time to generate one second of video and peak GPU memory, on one H100 with FlashAttention 3 at 512×768 (1280×768 for HD), from a warm run of one chunk (125 frames, 5 s). with compile is compile.strategy: torch (off by default); on top of these times, a run spends ~20 s compiling once at the start. 8 GPUs is the same generation split across 8 GPUs and counts DiT sampling alone.

Config	Time per 1 s of video	with `compile`	8 GPUs	Peak VRAM
`s2v_pro.yaml`	189 s	152 s	32 s	46 GiB
`s2v_pro_hd.yaml`	773 s	678 s	114 s	55 GiB

Peak memory is dominated by the VAE decode rather than the transformer; tiling the decode cuts the peak substantially at the cost of faint tile seams.

Responsible use

The model animates a real person from a single photograph, driven by a voice that need not be theirs. Use a reference frame you have the subject's consent to animate, and do not present the result as a genuine recording of anyone.

kandinskylab/Kandinsky-S2V-1.0-Pro