ComfyUI Node: Extract H3 RefMod

Extract H3 RefMod:

The MiniMaxH3RefModExtract node is designed to transform reference images or videos into a .safetensors mod that can be utilized in the MiniMax H3 generation process. This node is particularly useful for AI artists who want to incorporate specific visual references into their generative models without the need for extensive training. It operates in two modes: encode and training. The encode mode is ideal for capturing detailed identity features such as faces or outfits by resizing references to a specified resolution and encoding them, resulting in files that are approximately 0.2-1 MB per frame. The training mode, on the other hand, focuses on capturing broader concepts or motions by resizing and pooling references into a smaller grid, making it computationally efficient to inject but less detailed in terms of identity. This node leverages the H3 video VAE for encoding, ensuring that the process is streamlined and efficient without loading the larger 29B DiT model.

Extract H3 RefMod Input Parameters:

image

This parameter specifies the path to the reference image file that you want to encode. It is used in the encode mode to capture detailed identity features. The image is resized to the specified resolution's short edge before encoding. There are no explicit minimum or maximum values for this parameter, but the image should be of sufficient quality to capture the desired details.

video

This parameter specifies the path to the reference video file that you want to encode. It is used in both encode and training modes. In encode mode, the video frames are resized and encoded similarly to images. In training mode, the video is pooled into a smaller grid to capture broader concepts or motions. There are no explicit minimum or maximum values, but the video should be of good quality to ensure effective encoding.

vae

This parameter is the path to the H3 video VAE model file (.safetensors) required for encoding the references. It is essential for both modes as it provides the necessary encoding capabilities. There are no specific minimum or maximum values, but the correct VAE model must be used for successful encoding.

name

This parameter allows you to specify a name for the generated mod file. It is used to identify the mod when loading it into ComfyUI. There are no specific constraints on the name, but it should be descriptive enough to easily identify the mod's content.

mode

This parameter determines the mode of operation for the node. It can be set to either encode or training. The encode mode captures detailed identity features, while the training mode focuses on broader concepts or motions. The default value is encode.

resolution

This parameter sets the resolution to which the reference images or video frames are resized. It affects the level of detail captured in the encode mode and the computational efficiency in the training mode. There are no explicit minimum or maximum values, but a higher resolution captures more detail at the cost of increased file size.

pool

This parameter is used in the training mode to specify the size of the grid to which the references are pooled. The default value is 4, resulting in a 4x4 grid. This parameter affects the level of detail captured in the pooled representation.

latent-frames

This parameter is used in the training mode to specify the number of latent frames to be used. It affects the temporal resolution of the captured motion or concept. There are no explicit minimum or maximum values, but a higher number of frames captures more temporal detail.

Extract H3 RefMod Output Parameters:

mod

The output of the MiniMaxH3RefModExtract node is a .safetensors mod file that contains the encoded reference information. This mod can be loaded into ComfyUI and used to inject the reference into the MiniMax H3 generation process. The mod file encapsulates the captured identity or concept, depending on the mode used, and is essential for incorporating specific visual references into your generative models.

Extract H3 RefMod Usage Tips:

• Use the encode mode when you need to capture detailed identity features such as faces or specific outfits. This mode is ideal for character-based references.
• Opt for the training mode when you want to capture broader concepts or motions, such as dance moves, with minimal computational overhead.
• Ensure that your reference images or videos are of high quality to maximize the effectiveness of the encoding process.
• Choose an appropriate resolution that balances detail capture and file size, especially in the encode mode.

Extract H3 RefMod Common Errors and Solutions:

"reference has an empty frame"

• Explanation: This error occurs when the reference image or video has an invalid frame size, possibly due to incorrect resolution settings or corrupted files.
• Solution: Verify that the reference files are valid and of good quality. Adjust the resolution parameter to ensure that the frames are resized correctly.

"identity only applies to training mode"

• Explanation: This warning indicates that the identity parameter was set in encode mode, where it has no effect.
• Solution: Use the identity parameter only in training mode to refine the pooled representation. If you are in encode mode, this parameter should be ignored.

"one frame alone is tokens > budget"

• Explanation: This error occurs when the token count for a single frame exceeds the specified budget.
• Solution: Lower the ref_resolution or adjust the pool parameters to reduce the token count and fit within the budget.