Kandinsky S2V Animates Photos From Audio: a 10-Second Clip in 1.5 Minutes on an H100

Imagine you need a virtual presenter for a training course: you have one photo of an employee and a recorded voiceover, and you don't want to shoot new video. That is the task Kandinsky S2V 1.0 (speech-to-video) is built for.

Kandinsky Lab, a lab affiliated with Sber, has released it as open source. The model takes a photo and an audio track and produces a clip in which the person speaks, moves their hands and changes facial expression. The fastest version, Lite distill, renders 10 seconds of video in about 1.

5 minutes on an NVIDIA H100 (a server-grade card for training and running models). In the training data, 25.8% of the speech is Russian, which is rare for open models.

The author of a Telegram review calls S2V one of the most interesting open models of this type in recent times. Below: what it can do and what it costs to run.

What goes in and what comes out

The model needs three inputs: one reference photo, finished speech, and a text prompt describing how the person speaks and what emotions they show. Prompts can be written in English or Russian. The output is video at 25 frames per second in standard (SD) or high (HD) resolution, with the same aspect ratio as the source photo.

The whole figure is animated: facial expression, posture, gestures, hands. Many avatar models move only the lips and head; here the result looks livelier. It also works with more than just people: the examples include cartoon characters and animals.

The model splits long speech into 5-second chunks and stitches them together through "visual memory", so the character's appearance doesn't drift from one segment to the next. HD is generated with the same weights as SD, so no separate model had to be trained for it.

Russian got special attention: 25.8% of the training speech is Russian, and quality was additionally checked on a Russian-language set. That matters for local products, videos and training materials.

One limitation: the model does not generate audio. You need a finished recording or speech synthesis (TTS), so a production setup requires a "text → voice → video" pipeline.

Sizes, weights and hardware

There are two versions: Lite with 2B parameters and Pro with 21B. Each has distilled variants: lighter versions that run in 16 steps and without CFG (an extra pass that strengthens prompt adherence). According to the authors, they are 5.4 times faster than the base model and lose almost no quality. That "almost" is worth testing on your own tasks.

The video transformer (DiT) itself weighs 4.2 GB for Lite and 39.8 GB for Pro. But once you add Qwen2.5-Omni, CLIP and Hunyuan VAE, you have to download about 25 GB and 65 GB respectively.

GPU requirements are steep: a single NVIDIA card with at least 48 GB of memory, and 80 GB for HD. Setting offload.vae_tiling: true in the config drops peak usage to about 28 GB, but seams appear between image tiles in the frames. A 24 GB consumer card is still not enough.

Speed: how long you wait for a clip

Measurements on an H100 are given in seconds of compute per second of video:

• Lite distill (512×768): 8.7 seconds, so a 10-second clip takes about 1.5 minutes;
• Pro distill: 41.5 seconds, about 7 minutes per 10 seconds;
• Pro HD (1280×768): 137 seconds, about 23 minutes per 10 seconds.

Spreading the computation across several cards (4 for Lite, 8 for Pro) brings this down to 3.8 and 11.4 seconds per second of video. The wait gets shorter, but the GPU rental bill grows.

What to do with it

Lite distill suits prototypes: voiced characters, video replies from an assistant, localized training clips. It makes sense to keep Pro and HD for final renders, where detail matters and you can afford to wait. 23 minutes on an H100 for 10 seconds of HD is no longer interactive; it is batch processing, and you should work out the economics in advance.

For Russian-language products, the open weights matter in themselves: you can run the model on your own infrastructure without sending faces and voices to external services. You can try it in the demo on Hugging Face, the code is on GitHub, and the weights are published in the kandinskylab collection.