ComfyUI Node: AuK Prompt Enhance

AuK Prompt Enhance:

The AuKPromptEnhance node is designed to enhance audio prompts by leveraging the capabilities of the Qwen2.5-Omni-3B model. This node serves as a local prompt enhancer that can utilize an optional Automatic Speech Recognition (ASR) transcript to refine model instructions, calculate task-aware durations, and prepare audio levels for whisper tasks. It is particularly useful for tasks that require precise audio manipulation, such as voice description text-to-speech (TTS) and voice cloning. By mapping loose requests to specific AuK tasks, the node ensures that audio instructions are rendered accurately and efficiently, providing a seamless experience for users looking to enhance their audio content.

AuK Prompt Enhance Input Parameters:

encoder

The encoder parameter is an essential input that comes from the AuK Encoder Loader. It must include the language head, which is crucial for processing the audio instructions. If the encoder's checkpoint does not include the language head, you may need to re-run the conversion tool on the original Qwen directory. This parameter ensures that the node has the necessary components to perform its tasks effectively.

instruction

The instruction parameter allows you to input a loose request in any wording. This text input is used by the language model to determine the closest AuK task and fill its template. The default value is "make her sound excited and say welcome home," but you can customize it to fit your specific needs. This parameter is crucial for guiding the node in generating the desired audio output.

context

The context parameter is an optional input that provides additional context for the task. It can include the STRING output of the AuK Whisper Transcribe on the source audio. This extra information can help the node better understand the nuances of the task and improve the accuracy of the audio enhancement.

audio

The audio parameter is an optional input that serves as the source or reference audio. It is required for calculating the duration of edit tasks and preparing the audio for whisper tasks. By connecting the prepared audio output to the Instruction Encode audio, you ensure that the node can accurately process and enhance the audio content.

max_new_tokens

The max_new_tokens parameter sets the generation budget for the language model's answer. It determines the maximum number of new tokens that can be generated, with a default value of 256. The minimum value is 32, and the maximum is 1024. This parameter allows you to control the length and complexity of the generated audio instructions.

AuK Prompt Enhance Output Parameters:

text

The text output parameter provides the enhanced audio instruction text. This output is the result of the node's processing and reflects the refined instructions based on the input parameters and the selected AuK task. It is crucial for understanding the modifications made to the original audio prompt.

seconds

The seconds output parameter indicates the estimated duration of the generated audio instruction. This value is calculated based on the task and the input parameters, ensuring that the audio output aligns with the desired length and timing.

task

The task output parameter specifies the AuK task that was selected based on the input instruction. It provides insight into how the node interpreted the request and which task template was used to generate the audio instruction.

prepared

The prepared output parameter contains the processed audio data, including any adjustments made to the waveform and sample rate. This output is essential for ensuring that the audio content is ready for further processing or playback.

AuK Prompt Enhance Usage Tips:

• Ensure that the encoder includes the language head to avoid errors during processing.
• Use the instruction parameter to clearly define the desired audio modifications, as this will guide the node in selecting the appropriate AuK task.
• Provide additional context through the context parameter when available, as it can enhance the accuracy of the audio enhancement.
• Adjust the max_new_tokens parameter to control the complexity and length of the generated audio instructions, especially for tasks requiring detailed output.

AuK Prompt Enhance Common Errors and Solutions:

Instruction cannot be empty.

• Explanation: This error occurs when the instruction parameter is left blank.
• Solution: Ensure that you provide a valid instruction text to guide the node in selecting the appropriate AuK task.

Prompt Enhance audio contains NaN or Inf.

• Explanation: This error indicates that the audio input contains invalid values, such as NaN (Not a Number) or Inf (Infinity).
• Solution: Verify the integrity of the audio input and ensure that it does not contain any invalid values before processing.

Prompt Enhance audio must be a non-empty ComfyUI AUDIO value.

• Explanation: This error occurs when the audio input is empty or not in the correct format.
• Solution: Provide a valid audio input that meets the requirements of the node, ensuring it is a non-empty ComfyUI AUDIO value.

Prompt Enhance audio has an invalid sample rate.

• Explanation: This error indicates that the sample rate of the audio input is not a positive integer.
• Solution: Check the sample rate of the audio input and ensure it is a valid positive integer before processing.