ComfyUI Node: FL YuE2 · Gemini Music Captioner

FL YuE2 · Gemini Music Captioner:

The FL_YuE2_GeminiMusicCaptioner is a sophisticated node designed to transcribe and analyze audio recordings, specifically music, by leveraging Google's Gemini models. This node is capable of generating detailed captions that include style descriptions and lyrics transcription, providing a comprehensive understanding of the musical content. It is particularly beneficial for AI artists and developers who wish to integrate music analysis into their workflows, offering a seamless way to convert audio inputs into structured textual data. The node ensures that the transcriptions are complete and consistent, making it a reliable tool for music captioning tasks.

FL YuE2 · Gemini Music Captioner Input Parameters:

audio_directory

This parameter specifies the folder containing the audio files to be processed. It can be a path relative to ComfyUI/input or an absolute path. The recordings in this directory are sent to Google for captioning. The default value is an empty string, and it is crucial to ensure the path is correct to avoid processing errors.

model

This parameter allows you to select the Gemini model used for listening to each recording. Options include gemini-3.8-flash, gemini-3.1-pro-preview, and gemini-2.5-pro. The choice of model can affect the accuracy and detail of the captions, with each model offering different capabilities. The tooltip suggests that API usage may incur charges.

task

This parameter determines the type of captioning task to perform: both for style descriptions and lyrics transcription, style for style descriptions only, or lyrics for lyrics transcription only. Selecting the appropriate task ensures that the node focuses on the desired aspect of the audio content.

replace_existing

A boolean parameter that, when set to True, regenerates existing captions instead of reusing saved results. The default value is False, which helps in saving time and resources by avoiding redundant processing.

instructions

This parameter allows you to provide additional directions for the Gemini model, such as how to describe the genre or handle unclear vocals. It supports multiline input, enabling detailed instructions to be given to the model for more tailored captioning results.

api_key

This is a required parameter where you must enter your Google Gemini API key for the node to function. It is important to clear this field before sharing a workflow to prevent unauthorized access. The default value is an empty string.

concurrent_requests

This integer parameter controls the number of recordings captioned simultaneously, with a default value of 3. The minimum is 1, and the maximum is 8. Adjusting this value can help manage API rate limits and optimize processing speed.

FL YuE2 · Gemini Music Captioner Output Parameters:

ui

This output parameter provides a dictionary containing the results of the captioning process. It includes keys like caption_test or captions, which list the names of the generated captions. This output is essential for accessing and reviewing the captioning results.

result

This output parameter returns a tuple containing the result of the captioning operation. It provides detailed information about the processed audio files, including the directory and song data, which can be used for further analysis or integration into other workflows.

FL YuE2 · Gemini Music Captioner Usage Tips:

• Ensure that the audio_directory path is correct and accessible to avoid processing errors.
• Choose the appropriate model based on the complexity and requirements of your audio content to achieve the best results.
• Use the instructions parameter to provide specific guidance to the Gemini model, especially when dealing with unique or complex audio content.
• Adjust the concurrent_requests parameter based on your API quota and processing needs to optimize performance and avoid rate-limit errors.

FL YuE2 · Gemini Music Captioner Common Errors and Solutions:

"Enter a Google API key on the Gemini Music Captioner node"

• Explanation: This error occurs when the API key is missing or not provided.
• Solution: Ensure that you enter a valid Google Gemini API key in the api_key parameter before running the node.

"Gemini returned an incomplete caption schema"

• Explanation: This error indicates that the returned caption data does not match the expected schema.
• Solution: Verify that the audio input is complete and correctly formatted. If the issue persists, consider reviewing the audio content for any anomalies.

"concurrent_requests must be between 1 and 8"

• Explanation: This error arises when the concurrent_requests parameter is set outside the allowed range.
• Solution: Adjust the concurrent_requests value to be within the range of 1 to 8.

"Gemini did not transcribe the full recording; shorten the input into reviewed song sections and retry"

• Explanation: This error suggests that the transcription was incomplete due to the length or complexity of the audio.
• Solution: Break down the audio into smaller sections and retry the captioning process to ensure complete transcription.