ComfyUI Node: Apply VDN-H3 (MiniMax-H3 Hybrid Attention)

Apply VDN-H3 (MiniMax-H3 Hybrid Attention):

The ApplyVDNH3 node is designed to enhance the capabilities of the MiniMax-H3 diffusion model by integrating Video Delta Net hybrid attention (VDN-H3). This node applies a sophisticated attention mechanism that combines exact softmax attention for nearby frames with a linear Video Delta Attention branch for distant contexts. This hybrid approach allows for efficient processing of video data, maintaining high fidelity in areas of interest while optimizing computational resources for less critical regions. The node requires a VDN checkpoint and a MiniMax-H3 base model to function, ensuring that the model can leverage the strengths of both exact and approximate attention mechanisms. By doing so, it provides a balanced approach to video processing, enhancing both performance and quality.

Apply VDN-H3 (MiniMax-H3 Hybrid Attention) Input Parameters:

model

This parameter represents the MiniMax-H3 diffusion model that you wish to patch. It should be connected once between the model loader and the sampler, serving as the base model for the VDN-H3 application.

vdn_checkpoint

The VDN checkpoint is a directory under models/vdn that contains the linear-branch weights and specifications. It must match the loaded base model, such as a stage-dmd-* for an 8-step distilled model. This checkpoint is crucial for applying the VDN-H3 enhancements.

apply_turbo_adapter

This boolean parameter determines whether to apply the 'turbo' adapter when the checkpoint includes one, typically for an 8-step model. The default value is True. When enabled, it optimizes the model for faster processing, especially useful for models distilled from a 50-step process.

strength

This float parameter controls the adapter strength, with a default value of 1.0. It ranges from 0.0 to 2.0, allowing you to adjust the intensity of the VDN-H3 application. A value of 1.0 corresponds to the released model's default setting.

lora_mode

This parameter offers options between "bypass" and "merge," with "merge" as the default. It is essential for 8-step DMD checkpoints, ensuring the correct integration of the VDN-H3 enhancements with the base model.

branch_weights

This parameter can be set to "auto" or manually configured. It determines the weight distribution across different branches of the model, impacting how attention is allocated during processing.

attention_backend

This parameter specifies the backend used for attention calculations, influencing the performance and compatibility of the VDN-H3 application.

verbose

A boolean parameter that, when enabled, provides detailed logging information during the node's execution. This can be helpful for debugging and understanding the node's internal processes.

retain_buffers

This parameter controls whether to retain intermediate buffers during processing. It can be set to "auto," "on," or "off," affecting memory usage and performance.

Apply VDN-H3 (MiniMax-H3 Hybrid Attention) Output Parameters:

MODEL

The output is a modified version of the input MiniMax-H3 model, now enhanced with VDN-H3 hybrid attention. This output model is optimized for video processing, balancing high-quality attention for important frames with efficient handling of less critical areas. The enhanced model is ready for further processing or sampling, providing improved performance and quality in video applications.

Apply VDN-H3 (MiniMax-H3 Hybrid Attention) Usage Tips:

• Ensure that the VDN checkpoint matches the base model to avoid compatibility issues and achieve optimal results.
• Use the apply_turbo_adapter option for faster processing when working with 8-step models, but be mindful of the trade-off between speed and quality.
• Adjust the strength parameter to fine-tune the balance between the original model's characteristics and the VDN-H3 enhancements, depending on your specific needs.
• Enable verbose mode if you encounter issues or need to understand the node's processing steps in detail.

Apply VDN-H3 (MiniMax-H3 Hybrid Attention) Common Errors and Solutions:

"VDN checkpoint not found"

• Explanation: The specified VDN checkpoint directory is missing or incorrectly specified.
• Solution: Verify that the checkpoint directory exists under models/vdn and matches the expected naming convention for the base model.

"Incompatible model and checkpoint"

• Explanation: The loaded MiniMax-H3 model does not match the VDN checkpoint's specifications.
• Solution: Ensure that the VDN checkpoint corresponds to the correct stage and version of the MiniMax-H3 model you are using.

"Memory allocation failed"

• Explanation: Insufficient memory available to process the model with the current settings.
• Solution: Adjust the retain_buffers setting to "off" or reduce the strength parameter to lower memory usage.