kandinsky-s2v Introduction
kandinsky-s2v brings Kandinsky S2V 1.0 talking-avatar generation to ComfyUI. You provide one reference image and a speech recording, and the extension generates a video of the pictured subject speaking, with the recording included as the soundtrack.
This is useful when you want to turn a still portrait into a speaking character without manually animating every frame. For example, you can create a narrated character introduction, a talking-avatar clip, or a dialogue performance based on a recorded voice.
The extension supports two models:
-
Lite distilled — 2 billion parameters: the smaller, faster option in the supplied performance test.
-
Pro distilled — 21 billion parameters: a larger alternative that uses the same workflow nodes but requires more storage and processing time. Both models generate at 25 frames per second using one of three supported resolutions:
-
512 × 512: square
-
512 × 768: portrait
-
768 × 512: landscape Longer recordings are processed in connected 5-second sections, up to the speech encoder’s 300-second input limit. Each section continues from the final frames of the previous section.
Requirements: ComfyUI 0.38.0 or newer, Python 3.10 or newer, and a CUDA-capable GPU. ComfyUI manages model loading and movement between GPU memory and system memory.
How kandinsky-s2v Works
Think of the workflow as giving a digital performer three kinds of direction:
| Input | What it contributes | |---|---| | Reference image | The visual starting point for the subject and scene. | | Speech recording | The spoken performance that drives lip synchronization. | | Text prompt | A description of the scene and how the speaker should behave. |
The audio controls the lip-sync—not the words written in the prompt. If you want the subject to say something different, you need to change the recording.
1. Prepare the reference image
The Kandinsky S2V Reference Image node chooses the closest supported resolution based on your image’s aspect ratio.
It then:
- Resizes the image until it fills the chosen frame.
- Crops any excess from the center.
- Sends the prepared image to VAE Encode, which converts it into a compact representation the video model can use. This is similar to filling a picture frame: the image covers the frame completely, but parts near the edges may be cropped.
2. Interpret the speech and prompt
The extension uses Qwen2.5-Omni-7B to encode the speech and text prompt. It also uses CLIP-L to encode the prompt.
An encoder translates an input into numerical information that the generation model can understand. You do not need to prepare these representations yourself; the workflow nodes handle them.
The speech is padded with silence when necessary so that generation can work in complete 5-second sections.
3. Generate connected video sections
The selected Lite or Pro model generates the animation using the extension’s π-Flow sampler.
Each 5-second section contains 125 frames at 25 fps. For longer clips, the next section uses the final frames of the previous section as a continuation reference, rather than starting independently.
Both supported models were distilled for 16 sampling segments per section. Distillation means they were trained to perform generation with this compact sampling process.
4. Decode, trim, and save
After generation:
- VAE Decode converts the model’s compact output into visible frames.
- Kandinsky S2V Trim To Audio removes frames generated for the added silence.
- Create Video combines the frames with the speech recording.
- Save Video saves the result as an
.mp4.
kandinsky-s2v Features
Ready-to-use Lite and Pro workflows
You can start with either bundled template:
- Kandinsky S2V 1.0 Lite distilled
- Kandinsky S2V 1.0 Pro distilled In ComfyUI, open Workflow → Browse Templates, find the extension, and choose a workflow. Alternatively, drag one of the linked JSON files onto the canvas.
Each template includes its own example image and speech recording:
| Workflow | Reference image | Speech recording |
|---|---|---|
| Lite | kandinsky_s2v_reference.png | kandinsky_s2v_speech.wav |
| Pro | kandinsky_s2v_pro_reference.png | kandinsky_s2v_pro_speech.wav |
These example files are placed in ComfyUI/input/ on first start. Existing files with the same names are not overwritten.
To create your own clip:
- Open a bundled workflow.
- Make sure its models are available.
- Select your reference in Load Image.
- Select your recording in Load Audio.
- Edit the prompt to describe the scene and the speaker’s behavior.
- Run the workflow.
- View the saved
.mp4output. The workflows use the same processing nodes. Their main differences are the selected diffusion model and the bundled example inputs.
Scene and behavior prompting
The prompt describes the scene and the way the speaker should behave. The supplied default comes from the reference configuration.
For example, a scene description might be:
A speaker facing the camera, talking calmly with restrained movement.
Treat this as a direction for the performance, not a script. The spoken words and their timing still come from the audio.
The models have guidance built into their distilled design, so the sampler has:
- No negative prompt
- No CFG scale Controls familiar from other diffusion workflows therefore do not necessarily apply here.
Full-length generation or short previews
The chunks setting in Kandinsky S2V Encode controls how much of the recording is generated.
| Setting | Effect |
|---|---|
| chunks = 0 | Covers the whole recording, within the encoder’s input limit. |
| chunks = 1 | Generates only the beginning, up to one 5-second section. |
| chunks = 2 | Generates the beginning across two sections, up to 10 seconds. |
A shorter preview lets you check framing, motion, and the overall result before spending time on a full recording. If the selected portion ends with padded silence, the trimming node removes the corresponding extra frames.
Sampling control
The steps setting in Kandinsky S2V Sampler is the number of π-Flow segments per section.
The supported distilled models were trained at 16, making this the reference setting to use when you want behavior aligned with the supplied implementation. Do not assume that increasing it will improve the result in the same way as increasing steps in another image-generation workflow.
Automatic framing
The reference-image node fits your image to the closest trained aspect ratio. You do not need to calculate the resize dimensions yourself.
Because it uses a center crop, check images where the face or another important detail is close to an edge. For example, a very wide image containing a small, off-center speaker may lose important content during cropping. Reframing the source image before generation can help you control what remains visible.
Automatic memory management
ComfyUI loads and offloads model components as needed rather than keeping every component on the GPU at once.
In the author’s reported setup:
- Lite ran with ComfyUI limited to 24 GB of VRAM, unloading the approximately 17.5 GB speech encoder before sampling.
- Pro also ran within the same GPU-memory limit, using ComfyUI’s dynamic VRAM management to keep only part of its approximately 40 GB diffusion transformer on the GPU. These figures describe demonstrated configurations, not a guarantee of identical behavior or speed on every computer.
Workflow node guide
| Node | Purpose |
|---|---|
| Load Diffusion Model | Selects the supported Lite distilled or Pro distilled generation model. |
| Kandinsky S2V Encoder Loader | Loads the Qwen2.5-Omni Thinker component from models/text_encoders. |
| Load CLIP | Loads clip_l.safetensors; the supplied workflow uses the stable_diffusion type. |
| Kandinsky S2V Encode | Encodes speech and prompt, prepares complete 5-second sections, and applies the chunks selection. |
| Kandinsky S2V Reference Image | Resizes and center-crops the reference to a supported resolution. |
| VAE Encode | Converts the prepared reference into the model’s compact image representation. |
| Kandinsky S2V Sampler | Generates connected video sections with π-Flow sampling. |
| VAE Decode | Converts the generated representation into video frames. |
| Kandinsky S2V Trim To Audio | Removes padding-related frames and supplies the 25 fps frame rate. |
| Create Video / Save Video | Combines frames with the recording and saves the video. |
kandinsky-s2v Models
Supported generation models
| Model | Size | When to choose it | |---|---:|---| | Kandinsky S2V 1.0 Lite distilled | 2B parameters | Start here for previews, experimentation, or a smaller model download. It was faster in the supplied benchmark. | | Kandinsky S2V 1.0 Pro distilled | 21B parameters | Use when you want to evaluate the larger model and can accommodate its additional storage and processing time. |
Both models use the same supported resolutions, frame rate, and 16-segment sampling setup. To change models, select the corresponding checkpoint in Load Diffusion Model, or open its bundled workflow.
The supplied information does not establish that Pro will produce a better result for every input. Comparing the same image, audio, and prompt is the clearest way to judge which suits your project.
Shared supporting models
Both workflows also require:
| Model | Role | |---|---| | HunyuanVideo VAE | Converts between visible images/video frames and the compact representation used during generation. | | CLIP-L | Encodes the text prompt into a pooled text representation. | | Qwen2.5-Omni-7B | Encodes speech and text. Only its Thinker component is loaded; its Talker weights remain on disk. |
The Lite and Pro workflows share these components. Only their generation model—the diffusion transformer, or DiT—differs.
Downloading models from the workflow
Open a bundled workflow and either:
- Click Download models in its Models note, or
- Choose Kandinsky S2V → Kandinsky S2V — Download models from the top menu. The menu action downloads the models required by the open workflow, or asks which workflow’s models you want.
Approximate total download sizes are:
- Lite: 27 GB, including about 22 GB for Qwen.
- Pro: 66 GB, including about 43 GB for the Pro DiT. Existing model files are reused, including files found through configured extra model paths. Models are not downloaded automatically at startup.
Download sizes are disk-storage figures, not GPU-memory requirements.
Model links and expected locations
Paths below are relative to your ComfyUI folder.
| Model | Source | Expected location |
|---|---|---|
| Lite distilled DiT | Download weights | models/diffusion_models/kandinsky_s2v_lite_distill.safetensors |
| Pro distilled DiT | Download weights | models/diffusion_models/kandinsky_s2v_pro_distill.safetensors |
| HunyuanVideo VAE | Download weights | models/vae/hunyuan_video_vae_bf16.safetensors |
| CLIP-L | Download weights | models/text_encoders/clip_l.safetensors |
| Qwen2.5-Omni-7B | Model repository — the whole repository | models/text_encoders/Qwen2.5-Omni-7B/ |
The diffusion-transformer file from a corresponding distilled Diffusers bundle is also compatible:
text Kandinsky-S2V-1.0-{Lite,Pro}-distill-Diffusers/ └── transformer/diffusion_pytorch_model.safetensors
It is the same DiT file under a different name.
Unsupported variants
This extension does not support:
- Non-distilled Lite or Pro checkpoints; they require a different sampler.
- The Pro distilled HD grid described by
s2v_pro_distill_hd.yaml. Selecting an unsupported checkpoint causes the loader to reject it and report the differences.
Performance reference
On an H100, with ComfyUI limited to 24 GB of VRAM, the author measured the example workflow shortened to two sections, or 10 seconds:
| Model | Reported generation time, including model loading | |---|---:| | Lite distilled | 115 seconds | | Pro distilled | 470 seconds |
These are reference measurements, not expected timings for other GPUs. Generating a 10-second clip can take substantially longer than 10 seconds.
Troubleshooting kandinsky-s2v
A required model is missing
- Open the bundled workflow you intend to use.
- Click Download models in its Models note.
- Allow the required downloads to complete.
- Confirm that the workflow’s model loaders point to the expected files.
- For manually downloaded files, compare their locations with the model table above. For Qwen2.5-Omni-7B, the documented download is the whole repository, not a single weight file.
If a download requires access to gated or private weights, authenticate on the computer running ComfyUI with hf auth login. For a remotely hosted setup, this must happen on the server, not only on your local computer.
The diffusion checkpoint is rejected
- Check the file selected in Load Diffusion Model.
- Confirm that it is Lite distilled or Pro distilled.
- Do not substitute a non-distilled Lite or Pro checkpoint.
- If you use a Diffusers bundle, select its compatible
transformer/diffusion_pytorch_model.safetensorsfile.
The face or composition is cropped unexpectedly
- Check whether your image is square, portrait, or landscape.
- Remember that the extension resizes to cover the closest supported frame, then center-crops.
- Reframe your source image so the important subject remains near the center.
- Generate a one-section preview before rendering the full recording.
Only the beginning of the recording is generated
- Find
chunksin Kandinsky S2V Encode. - If it is set to a positive number, the workflow is intentionally generating only the beginning.
- Set it to
0to cover the full recording. - Keep the speech encoder’s 300-second limit in mind. For recordings beyond that limit, prepare shorter recordings rather than expecting one run to handle the entire track.
The generated words do not match the prompt
- Listen to the recording selected in Load Audio.
- Confirm that it contains the speech you want.
- Replace the recording if the wording needs to change.
- Use the prompt for scene and behavior direction. The prompt does not replace the audio as the source of lip synchronization.
There are extra frames after the speech
- Confirm that Kandinsky S2V Trim To Audio remains connected after decoding.
- Use its trimmed frames and supplied frame rate for Create Video.
- Keep the original speech recording connected as the video’s audio source, as shown in the bundled workflow. Generation uses complete 5-second sections; the trimming node removes the portion corresponding to added silence.
Generation is slow or runs out of GPU memory
- Start with Lite distilled rather than Pro.
- Test a short section using
chunks = 1. - Confirm that your setup meets the documented ComfyUI and CUDA requirements.
- Check that ComfyUI’s model offloading is functioning; the reported 24 GB runs rely on memory management.
- For Pro, note that the reported configuration relies on dynamic VRAM, rather than keeping the entire model on the GPU. Reducing the number of sections limits the amount of video generated, but does not make the model weights smaller. The supplied documentation does not establish a minimum VRAM requirement below the reported 24 GB configuration.
Results differ slightly from the reference implementation
Exact comparisons depend on matching inputs and computation settings.
The author reports close agreement with the reference implementation, with these qualifications:
- Bit-identical diffusion-model forward passes were verified under ComfyUI’s default PyTorch attention. Sage, xformers, or flash attention compute differently.
- ComfyUI’s Load CLIP uses the supplied CLIP-L weights in fp16, while the reference uses fp32, producing a small embedding difference.
- ComfyUI’s VAE Encode uses the mean of the reference-image representation; the reference implementation samples its VAE posterior instead. These differences do not automatically indicate an incorrectly configured workflow.
Frequently asked questions
Can I use a negative prompt or CFG scale? No. Guidance is distilled into the supported models, and this sampler does not expose those controls.
Can I generate at arbitrary resolutions or frame rates? The documented setup uses 25 fps at 512 × 512, 512 × 768, or 768 × 512. The Pro distilled HD grid is not supported. Can I use a CPU-only computer? The documented requirements specify a CUDA GPU. CPU-only operation is not a supported configuration described here.
Do I need separate copies of every supporting model for Lite and Pro? No. Both workflows share the VAE, CLIP-L, and Qwen components. Only the DiT differs.
Why are Qwen’s Talker weights downloaded if they are not used? The documented model download includes the whole Qwen repository. At runtime, the extension loads only the Thinker component; Talker weights stay on disk.
Is there a version-by-version changelog? The supplied documentation describes support for Kandinsky S2V 1.0 Lite distilled and Pro distilled, but does not include a version-by-version update history.
Learn More about kandinsky-s2v
-
Author’s Kandinsky S2V repository The reference implementation used for the extension’s compatibility and fidelity comparisons.
-
Lite example workflow A practical starting point for learning the workflow with the smaller model.
-
Pro example workflow The corresponding workflow for testing the larger model.
-
Lite distilled model repository and Pro distilled model repository Sources for the supported generation-model files and their accompanying model information.
-
Qwen2.5-Omni-7B Information about the shared speech and text encoder.
License
The extension uses the MIT license; see LICENSE.
The model weights have their own licenses. Review the relevant model terms separately before using generated work in a project, particularly for commercial use.
