Kandinsky S2V 1.0 Pro-distill — Diffusers format

Speech-to-video generation: an avatar video is produced from a reference frame and a speech track. The whole figure is animated, not only the head. The audio is a condition — the model does not generate sound, it animates the person in the reference frame to match the speech it is given.

This repository holds the pro distilled checkpoint laid out for Diffusers. The same weights in the native layout live in kandinskylab/Kandinsky-S2V-1.0-Pro-distill.

Contents

The bundle is self-contained: the Thinker is exported without the Talker of the full Omni checkpoint, and CLIP without its vision tower, since neither is used.

Usage

import librosa
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video, load_image

pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-distill-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000)  # a mono waveform at 16 kHz, for the model
speech, sample_rate = librosa.load("speech.wav", sr=None)  # the original track, added to the output video

video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
    video,
    fps=25,
    output_path="avatar.mp4",
    audio=torch.from_numpy(speech)[None],
    audio_sample_rate=sample_rate,
)

Pro does not fit an 80 GB card with every component resident: beside the 7B speech encoder the VAE decode of a chunk has nowhere to go. Offload through a ComponentsManager instead of pipe.to("cuda"); the margin keeps room for the VAE decode of a chunk, which brings the peak to about 50 GiB:

from diffusers import ComponentsManager

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-distill-Diffusers", components_manager=manager)
pipe.load_components(dtype=torch.bfloat16)
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB")

The pipeline and its blocks come from modular_model_index.json — the distilled blocks here, with the pi-Flow scheduler and 16 steps, and without a guider or a negative prompt.

Kandinsky S2V 1.0 is not part of a released Diffusers version yet; it lands as a modular pipeline.

Notes

pi-Flow distillation, 16 steps, guidance distilled into the weights (the distilled blocks have no guider and no negative prompt). The output head is widened 5 times; the scheduler splits that grid, which is why out_visual_dim is 80.

Output is 125 frames per chunk at 25 fps (5 s) in one of the 512 buckets, chosen from the reference frame's aspect ratio. Longer speech is generated chunk by chunk, each conditioned on the previous one through FramePack memory.

Responsible use

The model animates a real person from a single photograph, driven by a voice that need not be theirs. Use a reference frame you have the subject's consent to animate, and do not present the result as a genuine recording of anyone.