ComfyUI-FishAudioS2 Introduction
ComfyUI-FishAudioS2 is an advanced extension designed to integrate the Fish Audio S2 Pro text-to-speech (TTS) model into the ComfyUI environment. This extension allows AI artists to generate highly realistic and emotionally rich speech from text inputs. With support for over 83 languages and more than 1500 emotion tags, it offers unparalleled control over the prosody and emotion of the generated speech. Whether you're looking to clone a voice from a short audio sample or create multi-speaker dialogues, ComfyUI-FishAudioS2 provides the tools to bring your audio projects to life with natural and expressive speech synthesis.
How ComfyUI-FishAudioS2 Works
At its core, ComfyUI-FishAudioS2 leverages the Fish Audio S2 Pro model, which uses a sophisticated Dual-Autoregressive (Dual-AR) architecture. This architecture consists of two main components: a slow autoregressive model that predicts the primary semantic codebook and a fast autoregressive model that generates additional acoustic details. This design allows for high-quality audio generation with fine-grained control over speech characteristics. The model is trained on a vast dataset of over 10 million hours of audio, enabling it to produce speech that is both natural and emotionally nuanced. By using reinforcement learning for alignment, the model ensures that the generated speech aligns well with human expectations in terms of semantic accuracy and emotional expression.
ComfyUI-FishAudioS2 Features
- Zero-Shot Voice Cloning: Clone any voice using just 10-30 seconds of reference audio. This feature allows you to replicate the timbre and style of a speaker without additional training.
- Emotion and Prosody Control: Use over 1500 emotion tags to control the emotional tone and prosody of the speech. Tags like
[laugh],[whisper], and[excited]can be embedded directly in the text to influence the output. - Multilingual Support: Generate speech in 83 languages without the need for phoneme preprocessing. The model automatically detects and adapts to the language of the input text.
- Multi-Speaker Dialogue Synthesis: Create dialogues with multiple speakers in a single generation. Assign different voices to different parts of the text using speaker tags.
- Performance Optimization: Supports various precision modes (bf16, fp16, fp32) and attention mechanisms (SDPA, FlashAttention, SageAttention) to optimize performance on different hardware configurations.
- Smart Model Management: Automatically downloads and caches models from HuggingFace, ensuring that you always have the latest version without manual intervention.
ComfyUI-FishAudioS2 Models
ComfyUI-FishAudioS2 offers several model variants to suit different hardware capabilities:
- s2-pro: The full-precision model requiring approximately 24GB of VRAM. It offers the highest quality output with a speed of 15-17 iterations per second.
- s2-pro-fp8: A quantized model using FP8 precision, suitable for GPUs with 20GB+ VRAM. It provides a good balance between quality and performance.
- BNB INT8 and BNB NF4: These models use real-time quantization via bitsandbytes, allowing them to run on GPUs with as little as 16GB of VRAM. They offer a speed of 10-11 iterations per second.
Troubleshooting ComfyUI-FishAudioS2
If you encounter issues while using ComfyUI-FishAudioS2, here are some common problems and solutions:
- Model Download Issues: If the model fails to download automatically, you can manually download it from HuggingFace and place it in the
ComfyUI/models/fishaudioS2/directory. - Dependency Conflicts: Ensure that all required packages are installed. If you encounter errors related to
protobuf, reinstalldescript-audio-codecanddescript-audiotoolswith the--no-depsoption to avoid conflicts. - Performance Issues: For slow synthesis, try enabling
sage_attentionor using a GPU with CUDA support. Adjusting thechunk_lengthandprecisionsettings can also help optimize performance. - Voice Cloning Errors: Ensure that the reference audio is clear and within the recommended duration of 10-30 seconds. If using Whisper for transcription, manually provide the reference text if compatibility issues arise.
Learn More about ComfyUI-FishAudioS2
To further explore the capabilities of ComfyUI-FishAudioS2, consider visiting the following resources:
- Fish Audio Website (https://fish.audio/): Explore live demos and additional information about the Fish Audio S2 Pro model.
- Technical Report: Dive into the technical details and research behind the model.
- HuggingFace Model Page: Access the model files and documentation.
- Fish Audio Discord: Join the community to discuss and get support from other users and developers.
By leveraging these resources, you can enhance your understanding and make the most of the powerful features offered by ComfyUI-FishAudioS2.
