ComfyUI  >  Nodes  >  ComfyUI's ControlNet Auxiliary Preprocessors

ComfyUI Extension: ComfyUI's ControlNet Auxiliary Preprocessors

Repo Name

comfyui_controlnet_aux

Author
Fannovel16 (Account age: 3127 days)
Nodes
View all nodes (59)
Latest Updated
6/18/2024
Github Stars
1.6K

How to Install ComfyUI's ControlNet Auxiliary Preprocessors

Install this extension via the ComfyUI Manager by searching for  ComfyUI's ControlNet Auxiliary Preprocessors
  • 1. Click the Manager button in the main menu
  • 2. Select Custom Nodes Manager button
  • 3. Enter ComfyUI's ControlNet Auxiliary Preprocessors in the search bar
After installation, click the  Restart button to restart ComfyUI. Then, manually refresh your browser to clear the cache and access the updated list of nodes.

Visit ComfyUI Online for ready-to-use ComfyUI environment

  • Free trial available
  • High-speed GPU machines
  • 200+ preloaded models/nodes
  • Freedom to upload custom models/nodes
  • 50+ ready-to-run workflows
  • 100% private workspace with up to 200GB storage
  • Dedicated Support

Run ComfyUI Online

ComfyUI's ControlNet Auxiliary Preprocessors Description

ComfyUI's ControlNet Auxiliary Preprocessors reworks the original preprocessors using ControlNet auxiliary models, replacing most v1 preprocessors with v1.1 versions. It maintains compatibility with old workflows but advises against using the controlnet preprocessor to avoid conflicts.

ComfyUI's ControlNet Auxiliary Preprocessors Introduction

Welcome to the comfyui_controlnet_aux extension! This extension is designed to enhance your experience with ComfyUI by providing a set of plug-and-play nodes for generating hint images using ControlNet. These hint images can be used to guide the generation of AI art, making it easier to achieve the desired results. Whether you're looking to create detailed line art, depth maps, or pose estimations, comfyui_controlnet_aux has you covered.

How ComfyUI's ControlNet Auxiliary Preprocessors Works

At its core, comfyui_controlnet_aux leverages the power of ControlNet, a neural network structure that adds extra conditions to diffusion models. This allows you to control the output of your AI art generation more precisely. The extension works by copying the weights of neural network blocks into a "locked" copy and a "trainable" copy. The trainable copy learns your conditions, while the locked copy preserves the original model. This ensures that your original model remains intact while allowing for fine-tuning based on your specific needs.

ComfyUI's ControlNet Auxiliary Preprocessors Features

Line Extractors

  • Binary Lines: Extracts binary line art from images.
  • Canny Edge: Detects edges using the Canny edge detection algorithm.
  • HED Soft-Edge Lines: Generates soft-edge lines using HED.
  • Standard Lineart: Produces standard line art.
  • Realistic Lineart: Creates realistic line art, suitable for detailed illustrations.
  • Anime Lineart: Tailored for anime-style line art.
  • Manga Lineart: Ideal for manga-style illustrations.
  • M-LSD Lines: Detects straight lines using M-LSD.
  • PiDiNet Soft-Edge Lines: Generates soft-edge lines using PiDiNet.
  • Scribble Lines: Converts images into scribble-like lines.
  • TEED Soft-Edge Lines: Produces soft-edge lines using TEED.
  • AnyLine Lineart: A versatile line art extractor.

Normal and Depth Estimators

  • MiDaS Depth Map: Generates depth maps using MiDaS.
  • LeReS Depth Map: Produces depth maps using LeReS.
  • Zoe Depth Map: Creates depth maps using Zoe.
  • MiDaS Normal Map: Generates normal maps using MiDaS.
  • BAE Normal Map: Produces normal maps using BAE.
  • MeshGraphormer Hand Refiner: Refines hand meshes for 3D printing.
  • Depth Anything: A versatile depth map generator.
  • Normal DSINE: Generates normal maps using DSINE.
  • Metric3D Depth: Produces metric depth maps.
  • Metric3D Normal: Generates metric normal maps.

Faces and Poses Estimators

  • DWPose Estimator: Estimates human poses using DWPose.
  • OpenPose Estimator: Detects human poses using OpenPose.
  • MediaPipe Face Mesh: Generates face meshes using MediaPipe.
  • Animal Estimator: Detects animal poses.

Optical Flow Estimators

  • Unimatch Optical Flow: Estimates optical flow using Unimatch.

Semantic Segmentation

  • OneFormer ADE20K Segmentor: Segments images using ADE20K.
  • OneFormer COCO Segmentor: Segments images using COCO.
  • UniFormer Segmentor: A versatile segmentation tool.

T2IAdapter-only

  • Color Pallete: Generates color palettes.
  • Content Shuffle: Shuffles content for style transfer.

Recolor

  • Image Luminance: Adjusts image luminance.
  • Image Intensity: Adjusts image intensity.

ComfyUI's ControlNet Auxiliary Preprocessors Models

The extension supports various models, each tailored for specific tasks. Here are some examples:

  • ControlNet Models: These models are used for generating hint images based on different conditions like edges, depth, and poses.
  • T2I-Adapter Models: These models are designed for text-to-image adaptation, allowing for more control over the generated images.

Troubleshooting ComfyUI's ControlNet Auxiliary Preprocessors

Common Issues and Solutions

  1. Missing Nodes: If some nodes don't appear after installation, ensure that all dependencies are installed correctly. If the issue persists, create an issue on the with the log from the command line.
  2. Slow Performance: For DWPose/AnimalPose, using TorchScript or ONNXRuntime can significantly speed up processing. Refer to the installation instructions for details on setting this up.
  3. Incorrect Outputs: Ensure that the correct model and settings are being used for your specific task. Adjusting parameters like thresholds can also help improve results.

Frequently Asked Questions

  • Why are some nodes missing?
  • This could be due to missing dependencies or incorrect installation. Check the installation steps and ensure all requirements are met.
  • How can I speed up DWPose/AnimalPose?
  • Use TorchScript or ONNXRuntime for faster processing. Detailed instructions are provided in the troubleshooting section.

Learn More about ComfyUI's ControlNet Auxiliary Preprocessors

For more information, tutorials, and community support, check out the following resources:

  • These resources provide comprehensive guides, examples, and community forums where you can ask questions and get support from other AI artists and developers.

ComfyUI's ControlNet Auxiliary Preprocessors Related Nodes

RunComfy

© Copyright 2024 RunComfy. All Rights Reserved.

RunComfy is the premier ComfyUI platform, offering ComfyUI online environment and services, along with ComfyUI workflows featuring stunning visuals.