ComfyUI Node: Mouth Shape Compositor (Tracked)

Mouth Shape Compositor (Tracked):

The MouthShapeCompositorTracked node is designed to seamlessly integrate mouth shapes into video frames by utilizing tracked positions. This node is particularly beneficial for applications involving lip-syncing in animations or video editing, where precise alignment of mouth shapes with facial movements is crucial. By leveraging tracking data, this node ensures that mouth shapes are accurately placed and blended into each frame, enhancing the realism and synchronization of the animation. The node supports various tracking modes, including automatic, point-based, and mask-based tracking, allowing for flexibility in how the mouth shapes are applied. Additionally, it offers options for background removal and scaling, providing further customization to suit different artistic needs. Overall, the MouthShapeCompositorTracked node is a powerful tool for artists looking to create dynamic and lifelike animations with minimal manual intervention.

Mouth Shape Compositor (Tracked) Input Parameters:

frames

This parameter represents the video frames into which the mouth shapes will be composited. It is a tensor with dimensions (B, H, W, C), where B is the batch size, H is the height, W is the width, and C is the number of color channels. The frames serve as the canvas for the mouth shapes, and their quality and resolution directly impact the final output.

mouth_shapes

This parameter consists of the mouth images to be composited, structured as a tensor with dimensions (9, MH, MW, C). It includes nine distinct mouth shapes, each with its own height (MH), width (MW), and color channels (C). These shapes are used to represent different phonetic positions and are crucial for achieving accurate lip-syncing.

mouth_sequence

This is a list of integers indicating which mouth shape to use for each frame. The sequence aligns with the frames, dictating the specific mouth shape to apply at each point in the video. It is essential for synchronizing the mouth movements with the audio or intended speech.

tracking_mode

This string parameter determines the method of tracking used for placing the mouth shapes. Options include "auto", "points", or "masks". The choice of tracking mode affects how the node interprets and applies the tracking data, influencing the precision and style of the mouth shape placement.

tracking_masks

An optional tensor parameter with dimensions (B, H, W) that provides per-frame masks for tracking. These masks can guide the placement of mouth shapes by defining specific areas of interest within each frame. If not provided, the node will rely on other tracking data.

point_sequence

An optional list of tuples containing (x, y) coordinates for each frame. This parameter provides explicit tracking points that can be used to position the mouth shapes accurately. It is particularly useful when precise manual control over the placement is required.

remove_background

A boolean parameter that, when set to true, enables background removal using the BiRefNet method. This option is useful for isolating the subject from the background, ensuring that the mouth shapes blend seamlessly without unwanted artifacts.

bg_method

This string parameter specifies the method used for background removal. The default method is "birefnet", which is known for its effectiveness in separating the foreground from the background in video frames.

bg_color

A string parameter that defines the color used for color keying during background removal. The default value is "white", but it can be adjusted to match the predominant background color in the video for optimal results.

scale

A float parameter that determines the scale factor for the mouth shapes. The default value is 1.0, meaning no scaling. Adjusting this value allows for resizing the mouth shapes to better fit the subject's face in the video.

offset_x

An integer parameter that specifies the horizontal offset from the centroid for placing the mouth shapes. This allows for fine-tuning the position of the mouth shapes to achieve the desired alignment.

offset_y

An integer parameter that specifies the vertical offset from the centroid for placing the mouth shapes. Similar to offset_x, this parameter helps in adjusting the vertical alignment of the mouth shapes.

output_mouth_rgba

A boolean parameter that, when enabled, outputs the per-frame mouth shapes with an alpha channel. This option is useful for further compositing or editing tasks where transparency is required.

output_cropped_mouths

A boolean parameter that, when enabled, outputs the cropped mouth shapes with an alpha channel. This provides a focused view of the mouth shapes, which can be useful for detailed analysis or further processing.

Mouth Shape Compositor (Tracked) Output Parameters:

composited_frames

This output is a tensor containing the video frames with the mouth shapes composited onto them. It represents the final result of the node's processing, showcasing the integrated mouth shapes within the original video context.

mouth_rgba_frames

This output is a tensor of the per-frame mouth shapes with an alpha channel, providing a transparent overlay of the mouth shapes. It is useful for further compositing tasks or when additional editing is required.

cropped_mouths

This output is a tensor of the cropped mouth shapes with an alpha channel, offering a focused view of the mouth shapes. It is particularly useful for detailed analysis or when the mouth shapes need to be isolated from the rest of the frame.

Mouth Shape Compositor (Tracked) Usage Tips:

• Ensure that the mouth_shapes parameter contains exactly nine distinct shapes to avoid errors and achieve accurate lip-syncing.
• Utilize the tracking_mode parameter to select the most suitable tracking method for your project, whether it be automatic, point-based, or mask-based.
• Adjust the scale, offset_x, and offset_y parameters to fine-tune the placement and size of the mouth shapes for optimal alignment with the subject's face.
• Enable remove_background if the video has a complex background that might interfere with the mouth shapes, ensuring a cleaner compositing result.

Mouth Shape Compositor (Tracked) Common Errors and Solutions:

Expected 9 mouth shapes, got <number>

• Explanation: This error occurs when the mouth_shapes parameter does not contain exactly nine shapes, which is required for the node to function correctly.
• Solution: Ensure that the mouth_shapes tensor includes exactly nine distinct mouth shapes, each representing a different phonetic position.

IndexError: index out of range in self

• Explanation: This error may occur if the mouth_sequence contains indices that exceed the available mouth shapes.
• Solution: Verify that all indices in the mouth_sequence are within the range of 0 to 8, corresponding to the nine available mouth shapes.

ValueError: Invalid tracking mode

• Explanation: This error indicates that an unsupported value was provided for the tracking_mode parameter.
• Solution: Ensure that the tracking_mode is set to one of the supported options: "auto", "points", or "masks".