ComfyUI Node: VAE Merge (C2C)

VAE Merge (C2C):

The VAEMergeMEC node is designed to merge two or three Variational Autoencoder (VAE) checkpoints using a variety of sophisticated strategies. This node is particularly useful for AI artists who want to blend different VAE models to create unique and enhanced outputs. It offers 13 different merging strategies, such as weighted sum, sigmoid, geometric, and more, allowing for a high degree of customization in the merging process. The node also supports per-block weights and can automatically determine the optimal blending ratio (auto-alpha) based on block cosine similarity. Additionally, it includes a latent-space probe to assess reconstruction quality through metrics like Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR). The node is capable of performing dry-runs and exporting recipes for future use, making it a versatile tool for model experimentation and refinement.

VAE Merge (C2C) Input Parameters:

vae_a

This parameter represents the primary VAE model, which acts as the base for the merging process. It is deep-copied and serves as the foundation onto which other VAEs are blended. The quality and characteristics of this VAE significantly influence the final merged output.

vae_b

The secondary VAE model that is blended into vae_a. This VAE contributes additional features and characteristics to the merged model, allowing for creative combinations and enhancements. The blending process is influenced by the selected merge strategy.

vae_c

An optional third VAE model required for certain merge strategies like add_difference, triple_sum, and smooth_add_diff. When used, it provides additional complexity and depth to the merging process, enabling more intricate model combinations.

merge_mode

This parameter determines the strategy used to blend the VAEs. Options include weighted_sum, sigmoid, geometric, and others, each offering a unique method of combining the models. The choice of merge mode affects the final output's characteristics and should be selected based on the desired artistic effect.

alpha

A blending ratio that influences the weight given to vae_b during the merge. It can be automatically determined (auto-alpha) or manually set, allowing for precise control over the merging process. The value typically ranges from 0 to 1.

beta

An additional blending ratio used in certain merge strategies to further refine the merging process. Like alpha, it provides control over the influence of the secondary VAE and can be adjusted to achieve specific artistic goals.

use_blocks

A boolean parameter that, when enabled, allows for per-block weighting during the merge. This feature provides granular control over which parts of the VAEs are emphasized in the final output, enabling more targeted and creative model combinations.

VAE Merge (C2C) Output Parameters:

merged_vae

The resulting VAE model after the merge process. This output is a new VAE that combines the characteristics of the input models according to the selected merge strategy and parameters. It can be used for further artistic exploration or as a base for additional merges.

info

A JSON-formatted string containing detailed information about the merge process, including the architecture of the merged model, key counts, and any warnings or errors encountered. This output is useful for understanding the merge's technical aspects and troubleshooting any issues.

recipe_str

An exportable recipe string that captures the settings and parameters used in the merge. This allows for easy replication of the merge process in future projects or sharing with other artists.

probe_str

A JSON-formatted string containing the results of the latent-space probe, including metrics like MSE and PSNR. This output provides insights into the quality of the merged model's reconstruction capabilities, helping to assess its effectiveness.

VAE Merge (C2C) Usage Tips:

• Experiment with different merge_mode settings to discover unique combinations and effects. Each mode offers a distinct way of blending models, so try several to find the one that best suits your artistic vision.
• Use the auto-alpha feature to let the node determine the optimal blending ratio based on cosine similarity. This can save time and ensure a balanced merge without manual adjustments.
• Enable use_blocks for more control over the merging process. This allows you to emphasize specific parts of the VAEs, creating more nuanced and targeted outputs.

VAE Merge (C2C) Common Errors and Solutions:

" VAE Merge (C2C): required VAE input(s) missing: vae_a, vae_b."

• Explanation: This error occurs when one or both of the primary VAE inputs are not connected.
• Solution: Ensure that both vae_a and vae_b are properly connected to the node. These are essential inputs for the merging process.

"[MEC VAE Merge] 'merge_mode' requires vae_c to be connected."

• Explanation: Certain merge modes require a third VAE (vae_c) to function correctly.
• Solution: Connect a third VAE to the vae_c input or switch to a merge mode that does not require a third VAE, such as weighted_sum.

"load_state_dict failed: <exception>"

• Explanation: This error indicates a problem with loading the state dictionary of one of the VAEs, possibly due to incompatibility or corruption.
• Solution: Verify that the VAEs are compatible and not corrupted. Check the model files and ensure they are correctly formatted and accessible.