ComfyUI Node: CLIP Vision Multi-Ref Switch

CLIP Vision Multi-Ref Switch:

The CLIPVisionMultiRefSwitch node is designed to enhance the processing of CLIP vision outputs by enabling multi-reference support, particularly for use in Wan SCAIL models. This node plays a crucial role in scenarios where multiple reference images are involved, as it allows all images in a batch to contribute to the model's conditioning process. When activated, the node reshapes the batch dimension of the CLIP vision output into the token dimension, ensuring that all reference images are processed through the model's MLPProj layer, rather than just the first image. This capability is particularly beneficial for applications that require comprehensive image analysis and feature extraction from multiple images, thereby improving the model's performance and output quality. By placing this node between the CLIPVisionEncode and WanSCAILToVideoMultiRef or WanAnimateToVideoCustom, users can seamlessly integrate multi-image processing into their workflows.

CLIP Vision Multi-Ref Switch Input Parameters:

clip_vision_output

This parameter represents the output from the CLIPVisionEncode node and serves as the input to the CLIPVisionMultiRefSwitch. It supports a batch of images, allowing multiple images to be processed simultaneously. The function of this parameter is to provide the necessary data for the node to determine whether multi-image concatenation is required. There are no specific minimum or maximum values for this parameter, as it is dependent on the batch size of the input images.

enabled

The enabled parameter is a boolean switch that determines whether the multi-image CLIP feature concatenation is activated. When set to True, the node reshapes the penultimate hidden states of the CLIP vision output, allowing all images in the batch to contribute to the model's conditioning. Conversely, when set to False, the node passes the CLIP vision output through unchanged, using only the first image's features. The default value for this parameter is False, providing users with the flexibility to enable or disable the feature based on their specific needs.

CLIP Vision Multi-Ref Switch Output Parameters:

clip_vision_output

The output parameter, also named clip_vision_output, provides the processed CLIP vision output. If the enabled parameter is set to True, the penultimate hidden states are reshaped from a batch dimension of [N, 257, 1280] to a token dimension of [1, N*257, 1280]. This transformation ensures that all reference images in the batch are considered in the model's processing, enhancing the overall feature extraction and analysis. If enabled is False, the output remains unchanged from the input, maintaining the original structure and using only the first image's features.

CLIP Vision Multi-Ref Switch Usage Tips:

• To fully leverage the multi-image processing capabilities, ensure that the enabled parameter is set to True when working with batches of images that require comprehensive analysis.
• Place the CLIPVisionMultiRefSwitch node between CLIPVisionEncode and WanSCAILToVideoMultiRef or WanAnimateToVideoCustom to seamlessly integrate it into your workflow and optimize the conditioning process for video models.

CLIP Vision Multi-Ref Switch Common Errors and Solutions:

"clip_vision_output is None"

• Explanation: This error occurs when the input to the node is None, indicating that the CLIPVisionEncode node did not produce a valid output.
• Solution: Ensure that the CLIPVisionEncode node is correctly configured and producing a valid output before connecting it to the CLIPVisionMultiRefSwitch.

"penultimate_hidden_states is None or shape[0] <= 1"

• Explanation: This error suggests that the penultimate hidden states are either missing or the batch size is insufficient for multi-image processing.
• Solution: Verify that the input batch contains multiple images and that the CLIPVisionEncode node is correctly outputting the penultimate hidden states. Adjust the batch size if necessary to include more than one image.