ComfyUI Extension: ComfyUI-Woosh

ComfyUI-Woosh Introduction

ComfyUI-Woosh is an innovative extension designed to enhance your creative projects by generating sound effects from text descriptions or video inputs. Developed using Sony AI's Woosh foundation model, this extension integrates seamlessly with ComfyUI, allowing AI artists to create immersive audio experiences without needing extensive technical knowledge. Whether you're looking to add dynamic soundscapes to your digital art or transform video frames into audio, ComfyUI-Woosh provides a versatile and user-friendly solution.

How ComfyUI-Woosh Works

At its core, ComfyUI-Woosh leverages advanced generative models to convert text and video inputs into audio outputs. Imagine describing a scene with words, and the extension brings it to life with sound, or taking a silent video and giving it a voice. This is achieved through a process called latent diffusion modeling, where the extension interprets input data and generates corresponding audio. By using distilled models, ComfyUI-Woosh ensures fast and efficient sound generation, making it accessible even for those with limited computational resources.

ComfyUI-Woosh Features

• Text-to-Audio (T2A): Transform text descriptions into sound effects using Flow and DFlow models. This feature allows you to create audio that matches the mood and theme of your visual art.
• Video-to-Audio (V2A): Convert video frames into audio using VFlow and DVFlow models. This is perfect for adding soundtracks to animations or video projects.
• Distilled Models: DFlow and DVFlow models offer rapid audio generation with fewer steps, making them ideal for quick iterations and experimentation.
• Dynamic VRAM Management: Efficiently manage your system's resources by offloading tasks between GPU and CPU, ensuring smooth performance even on less powerful machines.
• Force Offload: Automatically clear models from memory after use, optimizing system performance for subsequent tasks.
• Video Output: Directly output video frames for further processing or combination with audio, streamlining your workflow.
• Bundled Library: The Woosh library is included, eliminating the need for additional installations and ensuring compatibility with your existing environment.

ComfyUI-Woosh Models

ComfyUI-Woosh offers several models tailored to different tasks:

• Flow: Ideal for high-quality text-to-audio generation, offering the best sound fidelity.
• DFlow: A distilled version of Flow, providing faster audio generation with slightly reduced quality, suitable for quick previews.
• VFlow: Designed for video-to-audio conversion, maintaining high audio quality from video inputs.
• DVFlow: A distilled version of VFlow, optimized for speed, making it perfect for rapid prototyping. Each model is designed to cater to specific needs, allowing you to choose based on your project's requirements and available resources.

Troubleshooting ComfyUI-Woosh

Here are some common issues you might encounter and how to resolve them:

• Error Loading State_dict in Strict Mode: This is normal and handled by non-strict loading. It occurs when some checkpoint keys don't match.
• RoBERTa/HuggingFace Downloads Every Restart: The first download is cached locally, so subsequent runs should use the cache.
• CUDA Out of Memory: Enable force_offload to free up memory after each run, use smaller models like DFlow/DVFlow, or reduce the number of latent_frames.
• Model Download Fails (China): Set a HuggingFace mirror before starting ComfyUI to ensure successful downloads.
• Import Errors After Install: Restart ComfyUI to reload all necessary Python modules.

Learn More about ComfyUI-Woosh

To further explore the capabilities of ComfyUI-Woosh, consider visiting the following resources:

• Hugging Face Woosh Models for downloading model checkpoints.
• SonyResearch/Woosh GitHub Repository for in-depth technical details and updates.
• ComfyUI-VideoHelperSuite for additional video processing tools that complement ComfyUI-Woosh. These resources provide valuable insights and community support, helping you make the most of ComfyUI-Woosh in your creative endeavors.