ComfyUI Extension: LoRA Optimizer

ComfyUI-LoRA-Optimizer Introduction

The ComfyUI-LoRA-Optimizer is a powerful extension designed to enhance the process of combining multiple LoRAs (Low-Rank Adaptations) in ComfyUI. When stacking multiple LoRAs, users often encounter issues such as oversaturation, artifacts, or loss of detail. This extension addresses these problems by automatically determining the optimal way to merge your LoRAs. It analyzes potential conflicts between LoRAs, resolves these conflicts at the model layer level, and adjusts the strengths to ensure a clean and coherent result. By using this optimizer, you can achieve high-quality outputs without the typical drawbacks of manual LoRA stacking.

How ComfyUI-LoRA-Optimizer Works

The ComfyUI-LoRA-Optimizer operates by analyzing the interactions between different LoRAs in a stack. It identifies where conflicts occur and applies a variety of strategies to resolve them. The optimizer uses a two-pass streaming architecture to minimize memory usage. In the first pass, it analyzes the LoRAs to gather data on conflicts and strengths. In the second pass, it applies the optimal merging strategy based on the analysis. This approach ensures that non-overlapping regions retain their full effect while conflicting regions are resolved using advanced techniques like TIES (Trim, Elect Sign, Disjoint Merge).

ComfyUI-LoRA-Optimizer Features

• Automatic Conflict Resolution: The optimizer automatically detects and resolves conflicts between LoRAs, ensuring a smooth and artifact-free output.
• Adaptive Merge Strategies: It applies different merging strategies based on the specific conditions of each model layer, such as weighted sum, weighted average, and TIES.
• Sparsification: Implements DARE and DELLA sparsification techniques to reduce parameter interference, preserving unique features of each LoRA.
• Merge Refinement: Offers optional preprocessing steps to enhance merge quality, including direction orthogonalization and TALL-mask protection.
• Auto-Strength Adjustment: Automatically adjusts the strength of each LoRA to prevent overexposure, especially useful for models with multiple LoRAs.
• Key Normalization: Ensures compatibility across different LoRA trainers by normalizing key naming conventions.
• Architecture-Aware Profiles: Tailors numeric thresholds and strategies based on the architecture family, ensuring optimal performance.

ComfyUI-LoRA-Optimizer Models

The optimizer supports a wide range of models, including SD 1.5, SDXL, Flux, Z-Image, Wan, LTX Video, and more. Each model benefits from architecture-specific optimizations, ensuring that the merging process is tailored to the unique characteristics of the model.

Troubleshooting ComfyUI-LoRA-Optimizer

If you encounter issues while using the ComfyUI-LoRA-Optimizer, consider the following solutions:

• Artifacts or Overexposure: Try adjusting the auto_strength and output_strength settings to balance the LoRA effects.
• Memory Issues: Enable free_vram_between_passes to reduce peak VRAM usage.
• Unexpected Results: Ensure that all LoRAs are compatible with the optimizer by checking their key naming conventions and ensuring they are supported by the architecture-aware key normalization.

Learn More about ComfyUI-LoRA-Optimizer

To further explore the capabilities of the ComfyUI-LoRA-Optimizer, you can access additional resources such as the Nodes wiki page, Configuration Guide, and Workflows. These resources provide detailed information on configuring and using the optimizer effectively. Additionally, community forums and tutorials can offer support and insights from other AI artists who have used the extension.