ComfyUI Node: Apply VDN-H3 Advanced (Ablations & Fast Kernels)

Apply VDN-H3 Advanced (Ablations & Fast Kernels):

The ApplyVDNH3Advanced node is designed to enhance the capabilities of the MiniMax-H3 diffusion model by integrating advanced features of the Video Delta Net hybrid attention (VDN-H3). This node allows you to apply per-adapter strengths, perform window, anchor, text, and branch ablations, and utilize compile-fused branch kernels for optimized performance. It is particularly beneficial for users looking to fine-tune their models with precise control over various parameters, ensuring that the defaults reproduce the released model exactly. The advanced features of this node provide flexibility and efficiency, making it a powerful tool for AI artists who want to experiment with different configurations and achieve high-quality results in video model patching.

Apply VDN-H3 Advanced (Ablations & Fast Kernels) Input Parameters:

model

This parameter represents the MiniMax-H3 diffusion model that you wish to patch. It should be chained once between the model loader and the sampler, serving as the base model for applying the VDN-H3 enhancements.

vdn_checkpoint

The vdn_checkpoint parameter specifies the directory containing the VDN stage, which holds the linear-branch weights and specifications. It is crucial that this directory matches the loaded base model, such as an 8-step distilled model, to ensure compatibility and optimal performance.

apply_turbo_adapter

This boolean parameter determines whether to apply the 'turbo' adapter when the checkpoint includes one. By default, it is set to True, which is recommended for 8-step models. This setting can significantly impact the model's performance, especially when dealing with different sampler steps.

stage_b_strength

This float parameter controls the strength of the 'default' (Stage-B) adapter, with a default value of 1.0. It ranges from 0.0 to 2.0, allowing you to adjust the influence of the Stage-B adapter on the model's output.

turbo_strength

Similar to stage_b_strength, this float parameter adjusts the strength of the 'turbo' (8-step DMD) adapter. It also ranges from 0.0 to 2.0, with a default value of 1.0, providing flexibility in tuning the turbo adapter's effect.

lora_mode

The lora_mode parameter offers options between "bypass" and "merge," with "merge" as the default. This setting is essential for 8-step DMD checkpoints, as it dictates how the model integrates the LoRA (Low-Rank Adaptation) components.

branch_weights

This parameter can be set to "auto" or a specific value, determining the weights for the linear branch. When set to "auto," the system automatically decides the best configuration based on available resources.

attention_backend

This parameter specifies the backend used for attention mechanisms within the model. It plays a crucial role in determining the efficiency and speed of the attention processes.

verbose

A boolean parameter that, when enabled, provides detailed logging information during the execution of the node. This can be helpful for debugging and understanding the internal workings of the model.

retain_buffers

This parameter controls whether to retain buffers during execution, with options like "auto" or "on." The "auto" setting allows the system to decide based on available memory, optimizing resource usage.

window_radius

An integer parameter that defines the radius of the window used in the attention mechanism. It influences how much context is considered from surrounding frames, with a default value of 1.

window_chunk

This integer parameter specifies the chunk size for processing windows in the attention mechanism, with a default value of 5. It affects the granularity of the attention process.

anchor_frames

This parameter determines which frames are used as anchors in the attention mechanism, with options like "both." It impacts the reference points for attention calculations.

text_state

A boolean parameter that enables or disables the inclusion of text state in the attention mechanism. When enabled, it allows the model to consider textual information during processing.

linear_branch

This boolean parameter controls whether the linear branch is enabled, affecting the model's architecture and processing flow. By default, it is set to True.

fast_kernels

A boolean parameter that, when enabled, compiles the branch's hot spots for faster execution. This can significantly improve performance, especially in resource-intensive tasks.

Apply VDN-H3 Advanced (Ablations & Fast Kernels) Output Parameters:

MODEL

The output of the ApplyVDNH3Advanced node is a patched MiniMax-H3 diffusion model. This model incorporates the advanced VDN-H3 features, allowing for enhanced video processing capabilities. The output model is optimized based on the input parameters, providing improved performance and flexibility for various video-related tasks.

Apply VDN-H3 Advanced (Ablations & Fast Kernels) Usage Tips:

• To achieve optimal performance, ensure that the vdn_checkpoint directory matches the loaded base model, especially when working with 8-step distilled models.
• Experiment with different stage_b_strength and turbo_strength values to find the best balance for your specific use case, as these parameters significantly influence the model's output.
• Utilize the fast_kernels option for tasks that require high computational efficiency, as it can greatly enhance processing speed.

Apply VDN-H3 Advanced (Ablations & Fast Kernels) Common Errors and Solutions:

"Checkpoint directory mismatch"

• Explanation: This error occurs when the vdn_checkpoint directory does not match the loaded base model.
• Solution: Ensure that the directory specified in vdn_checkpoint corresponds to the correct model version, such as an 8-step distilled model.

"Insufficient memory for auto configuration"

• Explanation: The system cannot automatically configure branch_weights or retain_buffers due to limited available memory.
• Solution: Manually set branch_weights and retain_buffers to specific values to bypass the automatic configuration and optimize memory usage.

"Invalid parameter value"

• Explanation: One or more input parameters have values outside their allowed range or options.
• Solution: Double-check the parameter values, ensuring they fall within the specified ranges or options, such as stage_b_strength and turbo_strength being between 0.0 and 2.0.