Save 4 hours! We auto-setup your workflow! Free!

Drop your workflow.json — we handle every dependency, custom node, and model. Just open the link and run.

Auto-Setup Workflow Json (Free) Now!
ComfyUI > Nodes > kandinsky-s2v

ComfyUI Extension: kandinsky-s2v

Repo Name

kandinsky-s2v

Author
kandinskylab (Account age: 362 days)
Nodes
View all nodes(5)
Latest Updated
2026-11-05
Github Stars
0.04K

How to Install kandinsky-s2v

Install this extension via the ComfyUI Manager by searching for kandinsky-s2v
  • 1. Click the Manager button in the main menu
  • 2. Select Custom Nodes Manager button
  • 3. Enter kandinsky-s2v in the search bar
After installation, click the Restart button to restart ComfyUI. Then, manually refresh your browser to clear the cache and access the updated list of nodes.

Visit ComfyUI Online for ready-to-use ComfyUI environment

  • Free trial available
  • 16GB VRAM to 80GB VRAM GPU machines
  • 400+ preloaded models/nodes
  • Freedom to upload custom models/nodes
  • 200+ ready-to-run workflows
  • 100% private workspace with up to 200GB storage
  • Dedicated Support

Run ComfyUI Online

kandinsky-s2v Description

kandinsky-s2v generates talking-avatar videos in ComfyUI from a reference image and speech recording, using Kandinsky S2V models to synchronize the subject with the supplied audio.

kandinsky-s2v Introduction

kandinsky-s2v brings Kandinsky S2V 1.0 talking-avatar generation to ComfyUI. You provide one reference image and a speech recording, and the extension generates a video of the pictured subject speaking, with the recording included as the soundtrack.

This is useful when you want to turn a still portrait into a speaking character without manually animating every frame. For example, you can create a narrated character introduction, a talking-avatar clip, or a dialogue performance based on a recorded voice.

The extension supports two models:

  • Lite distilled — 2 billion parameters: the smaller, faster option in the supplied performance test.

  • Pro distilled — 21 billion parameters: a larger alternative that uses the same workflow nodes but requires more storage and processing time. Both models generate at 25 frames per second using one of three supported resolutions:

  • 512 × 512: square

  • 512 × 768: portrait

  • 768 × 512: landscape Longer recordings are processed in connected 5-second sections, up to the speech encoder’s 300-second input limit. Each section continues from the final frames of the previous section.

Requirements: ComfyUI 0.38.0 or newer, Python 3.10 or newer, and a CUDA-capable GPU. ComfyUI manages model loading and movement between GPU memory and system memory.

How kandinsky-s2v Works

Think of the workflow as giving a digital performer three kinds of direction:

| Input | What it contributes | |---|---| | Reference image | The visual starting point for the subject and scene. | | Speech recording | The spoken performance that drives lip synchronization. | | Text prompt | A description of the scene and how the speaker should behave. |

The audio controls the lip-sync—not the words written in the prompt. If you want the subject to say something different, you need to change the recording.

1. Prepare the reference image

The Kandinsky S2V Reference Image node chooses the closest supported resolution based on your image’s aspect ratio.

It then:

  1. Resizes the image until it fills the chosen frame.
  2. Crops any excess from the center.
  3. Sends the prepared image to VAE Encode, which converts it into a compact representation the video model can use. This is similar to filling a picture frame: the image covers the frame completely, but parts near the edges may be cropped.

2. Interpret the speech and prompt

The extension uses Qwen2.5-Omni-7B to encode the speech and text prompt. It also uses CLIP-L to encode the prompt.

An encoder translates an input into numerical information that the generation model can understand. You do not need to prepare these representations yourself; the workflow nodes handle them.

The speech is padded with silence when necessary so that generation can work in complete 5-second sections.

3. Generate connected video sections

The selected Lite or Pro model generates the animation using the extension’s π-Flow sampler.

Each 5-second section contains 125 frames at 25 fps. For longer clips, the next section uses the final frames of the previous section as a continuation reference, rather than starting independently.

Both supported models were distilled for 16 sampling segments per section. Distillation means they were trained to perform generation with this compact sampling process.

4. Decode, trim, and save

After generation:

  1. VAE Decode converts the model’s compact output into visible frames.
  2. Kandinsky S2V Trim To Audio removes frames generated for the added silence.
  3. Create Video combines the frames with the speech recording.
  4. Save Video saves the result as an .mp4.

kandinsky-s2v Features

Ready-to-use Lite and Pro workflows

You can start with either bundled template:

  • Kandinsky S2V 1.0 Lite distilled
  • Kandinsky S2V 1.0 Pro distilled In ComfyUI, open Workflow → Browse Templates, find the extension, and choose a workflow. Alternatively, drag one of the linked JSON files onto the canvas.

Each template includes its own example image and speech recording:

| Workflow | Reference image | Speech recording | |---|---|---| | Lite | kandinsky_s2v_reference.png | kandinsky_s2v_speech.wav | | Pro | kandinsky_s2v_pro_reference.png | kandinsky_s2v_pro_speech.wav |

These example files are placed in ComfyUI/input/ on first start. Existing files with the same names are not overwritten.

To create your own clip:

  1. Open a bundled workflow.
  2. Make sure its models are available.
  3. Select your reference in Load Image.
  4. Select your recording in Load Audio.
  5. Edit the prompt to describe the scene and the speaker’s behavior.
  6. Run the workflow.
  7. View the saved .mp4 output. The workflows use the same processing nodes. Their main differences are the selected diffusion model and the bundled example inputs.

Scene and behavior prompting

The prompt describes the scene and the way the speaker should behave. The supplied default comes from the reference configuration.

For example, a scene description might be:

A speaker facing the camera, talking calmly with restrained movement.

Treat this as a direction for the performance, not a script. The spoken words and their timing still come from the audio.

The models have guidance built into their distilled design, so the sampler has:

  • No negative prompt
  • No CFG scale Controls familiar from other diffusion workflows therefore do not necessarily apply here.

Full-length generation or short previews

The chunks setting in Kandinsky S2V Encode controls how much of the recording is generated.

| Setting | Effect | |---|---| | chunks = 0 | Covers the whole recording, within the encoder’s input limit. | | chunks = 1 | Generates only the beginning, up to one 5-second section. | | chunks = 2 | Generates the beginning across two sections, up to 10 seconds. |

A shorter preview lets you check framing, motion, and the overall result before spending time on a full recording. If the selected portion ends with padded silence, the trimming node removes the corresponding extra frames.

Sampling control

The steps setting in Kandinsky S2V Sampler is the number of π-Flow segments per section.

The supported distilled models were trained at 16, making this the reference setting to use when you want behavior aligned with the supplied implementation. Do not assume that increasing it will improve the result in the same way as increasing steps in another image-generation workflow.

Automatic framing

The reference-image node fits your image to the closest trained aspect ratio. You do not need to calculate the resize dimensions yourself.

Because it uses a center crop, check images where the face or another important detail is close to an edge. For example, a very wide image containing a small, off-center speaker may lose important content during cropping. Reframing the source image before generation can help you control what remains visible.

Automatic memory management

ComfyUI loads and offloads model components as needed rather than keeping every component on the GPU at once.

In the author’s reported setup:

  • Lite ran with ComfyUI limited to 24 GB of VRAM, unloading the approximately 17.5 GB speech encoder before sampling.
  • Pro also ran within the same GPU-memory limit, using ComfyUI’s dynamic VRAM management to keep only part of its approximately 40 GB diffusion transformer on the GPU. These figures describe demonstrated configurations, not a guarantee of identical behavior or speed on every computer.

Workflow node guide

| Node | Purpose | |---|---| | Load Diffusion Model | Selects the supported Lite distilled or Pro distilled generation model. | | Kandinsky S2V Encoder Loader | Loads the Qwen2.5-Omni Thinker component from models/text_encoders. | | Load CLIP | Loads clip_l.safetensors; the supplied workflow uses the stable_diffusion type. | | Kandinsky S2V Encode | Encodes speech and prompt, prepares complete 5-second sections, and applies the chunks selection. | | Kandinsky S2V Reference Image | Resizes and center-crops the reference to a supported resolution. | | VAE Encode | Converts the prepared reference into the model’s compact image representation. | | Kandinsky S2V Sampler | Generates connected video sections with π-Flow sampling. | | VAE Decode | Converts the generated representation into video frames. | | Kandinsky S2V Trim To Audio | Removes padding-related frames and supplies the 25 fps frame rate. | | Create Video / Save Video | Combines frames with the recording and saves the video. |

kandinsky-s2v Models

Supported generation models

| Model | Size | When to choose it | |---|---:|---| | Kandinsky S2V 1.0 Lite distilled | 2B parameters | Start here for previews, experimentation, or a smaller model download. It was faster in the supplied benchmark. | | Kandinsky S2V 1.0 Pro distilled | 21B parameters | Use when you want to evaluate the larger model and can accommodate its additional storage and processing time. |

Both models use the same supported resolutions, frame rate, and 16-segment sampling setup. To change models, select the corresponding checkpoint in Load Diffusion Model, or open its bundled workflow.

The supplied information does not establish that Pro will produce a better result for every input. Comparing the same image, audio, and prompt is the clearest way to judge which suits your project.

Shared supporting models

Both workflows also require:

| Model | Role | |---|---| | HunyuanVideo VAE | Converts between visible images/video frames and the compact representation used during generation. | | CLIP-L | Encodes the text prompt into a pooled text representation. | | Qwen2.5-Omni-7B | Encodes speech and text. Only its Thinker component is loaded; its Talker weights remain on disk. |

The Lite and Pro workflows share these components. Only their generation model—the diffusion transformer, or DiT—differs.

Downloading models from the workflow

Open a bundled workflow and either:

  • Click Download models in its Models note, or
  • Choose Kandinsky S2V → Kandinsky S2V — Download models from the top menu. The menu action downloads the models required by the open workflow, or asks which workflow’s models you want.

Approximate total download sizes are:

  • Lite: 27 GB, including about 22 GB for Qwen.
  • Pro: 66 GB, including about 43 GB for the Pro DiT. Existing model files are reused, including files found through configured extra model paths. Models are not downloaded automatically at startup.

Download sizes are disk-storage figures, not GPU-memory requirements.

Paths below are relative to your ComfyUI folder.

| Model | Source | Expected location | |---|---|---| | Lite distilled DiT | Download weights | models/diffusion_models/kandinsky_s2v_lite_distill.safetensors | | Pro distilled DiT | Download weights | models/diffusion_models/kandinsky_s2v_pro_distill.safetensors | | HunyuanVideo VAE | Download weights | models/vae/hunyuan_video_vae_bf16.safetensors | | CLIP-L | Download weights | models/text_encoders/clip_l.safetensors | | Qwen2.5-Omni-7B | Model repository — the whole repository | models/text_encoders/Qwen2.5-Omni-7B/ |

The diffusion-transformer file from a corresponding distilled Diffusers bundle is also compatible:

text Kandinsky-S2V-1.0-{Lite,Pro}-distill-Diffusers/ └── transformer/diffusion_pytorch_model.safetensors

It is the same DiT file under a different name.

Unsupported variants

This extension does not support:

  • Non-distilled Lite or Pro checkpoints; they require a different sampler.
  • The Pro distilled HD grid described by s2v_pro_distill_hd.yaml. Selecting an unsupported checkpoint causes the loader to reject it and report the differences.

Performance reference

On an H100, with ComfyUI limited to 24 GB of VRAM, the author measured the example workflow shortened to two sections, or 10 seconds:

| Model | Reported generation time, including model loading | |---|---:| | Lite distilled | 115 seconds | | Pro distilled | 470 seconds |

These are reference measurements, not expected timings for other GPUs. Generating a 10-second clip can take substantially longer than 10 seconds.

Troubleshooting kandinsky-s2v

A required model is missing

  1. Open the bundled workflow you intend to use.
  2. Click Download models in its Models note.
  3. Allow the required downloads to complete.
  4. Confirm that the workflow’s model loaders point to the expected files.
  5. For manually downloaded files, compare their locations with the model table above. For Qwen2.5-Omni-7B, the documented download is the whole repository, not a single weight file.

If a download requires access to gated or private weights, authenticate on the computer running ComfyUI with hf auth login. For a remotely hosted setup, this must happen on the server, not only on your local computer.

The diffusion checkpoint is rejected

  1. Check the file selected in Load Diffusion Model.
  2. Confirm that it is Lite distilled or Pro distilled.
  3. Do not substitute a non-distilled Lite or Pro checkpoint.
  4. If you use a Diffusers bundle, select its compatible transformer/diffusion_pytorch_model.safetensors file.

The face or composition is cropped unexpectedly

  1. Check whether your image is square, portrait, or landscape.
  2. Remember that the extension resizes to cover the closest supported frame, then center-crops.
  3. Reframe your source image so the important subject remains near the center.
  4. Generate a one-section preview before rendering the full recording.

Only the beginning of the recording is generated

  1. Find chunks in Kandinsky S2V Encode.
  2. If it is set to a positive number, the workflow is intentionally generating only the beginning.
  3. Set it to 0 to cover the full recording.
  4. Keep the speech encoder’s 300-second limit in mind. For recordings beyond that limit, prepare shorter recordings rather than expecting one run to handle the entire track.

The generated words do not match the prompt

  1. Listen to the recording selected in Load Audio.
  2. Confirm that it contains the speech you want.
  3. Replace the recording if the wording needs to change.
  4. Use the prompt for scene and behavior direction. The prompt does not replace the audio as the source of lip synchronization.

There are extra frames after the speech

  1. Confirm that Kandinsky S2V Trim To Audio remains connected after decoding.
  2. Use its trimmed frames and supplied frame rate for Create Video.
  3. Keep the original speech recording connected as the video’s audio source, as shown in the bundled workflow. Generation uses complete 5-second sections; the trimming node removes the portion corresponding to added silence.

Generation is slow or runs out of GPU memory

  1. Start with Lite distilled rather than Pro.
  2. Test a short section using chunks = 1.
  3. Confirm that your setup meets the documented ComfyUI and CUDA requirements.
  4. Check that ComfyUI’s model offloading is functioning; the reported 24 GB runs rely on memory management.
  5. For Pro, note that the reported configuration relies on dynamic VRAM, rather than keeping the entire model on the GPU. Reducing the number of sections limits the amount of video generated, but does not make the model weights smaller. The supplied documentation does not establish a minimum VRAM requirement below the reported 24 GB configuration.

Results differ slightly from the reference implementation

Exact comparisons depend on matching inputs and computation settings.

The author reports close agreement with the reference implementation, with these qualifications:

  • Bit-identical diffusion-model forward passes were verified under ComfyUI’s default PyTorch attention. Sage, xformers, or flash attention compute differently.
  • ComfyUI’s Load CLIP uses the supplied CLIP-L weights in fp16, while the reference uses fp32, producing a small embedding difference.
  • ComfyUI’s VAE Encode uses the mean of the reference-image representation; the reference implementation samples its VAE posterior instead. These differences do not automatically indicate an incorrectly configured workflow.

Frequently asked questions

Can I use a negative prompt or CFG scale? No. Guidance is distilled into the supported models, and this sampler does not expose those controls.

Can I generate at arbitrary resolutions or frame rates? The documented setup uses 25 fps at 512 × 512, 512 × 768, or 768 × 512. The Pro distilled HD grid is not supported. Can I use a CPU-only computer? The documented requirements specify a CUDA GPU. CPU-only operation is not a supported configuration described here.

Do I need separate copies of every supporting model for Lite and Pro? No. Both workflows share the VAE, CLIP-L, and Qwen components. Only the DiT differs.

Why are Qwen’s Talker weights downloaded if they are not used? The documented model download includes the whole Qwen repository. At runtime, the extension loads only the Thinker component; Talker weights stay on disk.

Is there a version-by-version changelog? The supplied documentation describes support for Kandinsky S2V 1.0 Lite distilled and Pro distilled, but does not include a version-by-version update history.

Learn More about kandinsky-s2v

License

The extension uses the MIT license; see LICENSE.

The model weights have their own licenses. Review the relevant model terms separately before using generated work in a project, particularly for commercial use.

kandinsky-s2v Related Nodes

RunComfy
Copyright 2025 RunComfy. All Rights Reserved.

RunComfy is the premier ComfyUI platform, offering ComfyUI online environment and services, along with ComfyUI workflows featuring stunning visuals. RunComfy also provides AI Models, enabling artists to harness the latest AI tools to create incredible art.

kandinsky-s2v detailed guide | ComfyUI