Kandinsky S2V 1.0 Pro distilled

Speech-to-video generation: an avatar video is produced from a reference frame and a speech track. The whole figure is animated, not only the head. The audio is a condition — the model does not generate sound, it animates the person in the reference frame to match the speech it is given.

This is the distilled variant: the sampler is pi-Flow, 16 steps, with classifier-free guidance distilled into the weights, so no negative pass is run at inference. It is roughly 5x faster than the 50-step base model.

Examples

Each is one reference frame plus one speech track. More examples — other languages, cartoon characters and animals, long videos — are on the project page.

Pipeline

Latent diffusion with flow matching.

• Qwen2.5-Omni-7B supplies both conditions: Thinker hidden states for the text prompt, and audio-tower features (3584-dim, 25 tokens/s) for the speech.
• CLIP ViT-L/14 supplies a pooled text embedding.
• HunyuanVideo 3D VAE encodes and decodes video.
• DiT (21.4B parameters, 60 visual blocks) is the generative backbone. Speech enters through per-layer cross-attention adapters on 18 of the visual blocks, frame-aligned so each latent frame attends only to its own audio tokens. FramePack memory at three temporal scales carries context from one chunk to the next.

Related

Kandinsky-S2V-1.0-Pro — the 50-step base model.

Contents

Text encoders and the VAE are not part of this repository; the code fetches them from their own repositories on first run:

Output

25 fps in SD or, with the *_hd.yaml configurations, HD resolution, at the reference frame's aspect ratio. Long speech is generated in 5-second chunks.

Usage

The code is in kandinskylab/kandinsky-s2v:

git clone https://github.com/kandinskylab/kandinsky-s2v
cd kandinsky-s2v
uv sync
uv run kandy-s2v generate \
    --config src/kandinsky_s2v/configs/s2v_pro_distill.yaml \
    --audio speech.wav \
    --image reference.png \
    --out out.mp4

The first run downloads this checkpoint, the encoders and the VAE from the Hub. For HD, pass src/kandinsky_s2v/configs/s2v_pro_distill_hd.yaml instead; it uses the same weights. To speed a generation up, split it across the GPUs of one node — see the repository README.

Performance

Time to generate one second of video and peak GPU memory, on one H100 with FlashAttention 3 at 512×768 (1280×768 for HD), from a warm run of one chunk (125 frames, 5 s). with compile is compile.strategy: torch (off by default); on top of these times, a run spends ~20 s compiling once at the start. 8 GPUs is the same generation split across 8 GPUs and counts DiT sampling alone.

Peak memory is dominated by the VAE decode rather than the transformer; tiling the decode cuts the peak substantially at the cost of faint tile seams.

Responsible use

The model animates a real person from a single photograph, driven by a voice that need not be theirs. Use a reference frame you have the subject's consent to animate, and do not present the result as a genuine recording of anyone.