Kandinsky S2V 1.0 Lite distilled

Speech-to-video generation: an avatar video is produced from a reference frame and a speech track. The whole figure is animated, not only the head. The audio is a condition — the model does not generate sound, it animates the person in the reference frame to match the speech it is given.

This is the distilled variant: the sampler is pi-Flow, 16 steps, with classifier-free guidance distilled into the weights, so no negative pass is run at inference. It is roughly 5x faster than the 50-step base model.

Examples

Each is one reference frame plus one speech track. More examples — other languages, cartoon characters and animals, long videos — are on the project page.

Pipeline

Latent diffusion with flow matching.

• Qwen2.5-Omni-7B supplies both conditions: Thinker hidden states for the text prompt, and audio-tower features (3584-dim, 25 tokens/s) for the speech.
• CLIP ViT-L/14 supplies a pooled text embedding.
• HunyuanVideo 3D VAE encodes and decodes video.
• DiT (2.3B parameters, 32 visual blocks) is the generative backbone. Speech enters through per-layer cross-attention adapters on 13 of the visual blocks, frame-aligned so each latent frame attends only to its own audio tokens. FramePack memory at three temporal scales carries context from one chunk to the next.

Related

Kandinsky-S2V-1.0-Lite — the 50-step base model.

Contents

Text encoders and the VAE are not part of this repository; the code fetches them from their own repositories on first run:

Output

25 fps in SD resolution, at the reference frame's aspect ratio. Long speech is generated in 5-second chunks.

Usage

The code is in kandinskylab/kandinsky-s2v:

git clone https://github.com/kandinskylab/kandinsky-s2v
cd kandinsky-s2v
uv sync
uv run kandy-s2v generate \
    --config src/kandinsky_s2v/configs/s2v_lite_distill.yaml \
    --audio speech.wav \
    --image reference.png \
    --out out.mp4

The first run downloads this checkpoint, the encoders and the VAE from the Hub. To speed a generation up, split it across the GPUs of one node — see the repository README.

Performance

Time to generate one second of video and peak GPU memory, on one H100 with FlashAttention 3 at 512×768, from a warm run of one chunk (125 frames, 5 s). with compile is compile.strategy: torch (off by default); on top of these times, a run spends ~20 s compiling once at the start. 4 GPUs is the same generation split across 4 GPUs and counts DiT sampling alone.

Peak memory is dominated by the VAE decode rather than the transformer; tiling the decode cuts the peak substantially at the cost of faint tile seams.

Responsible use

The model animates a real person from a single photograph, driven by a voice that need not be theirs. Use a reference frame you have the subject's consent to animate, and do not present the result as a genuine recording of anyone.