Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 Introduction
Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 adds native ComfyUI nodes for Alibaba PAI’s Qwen Image 2.1 Fun ControlNet Union. It helps you guide image generation and editing with visual information—not just text.
A text prompt describes what you want, but it may not reliably place a person, preserve a room’s layout, or follow a sketch. This extension lets you supply a control map: an image that describes structure, such as outlines, depth, or body pose. You can also provide photographs as editing references and masks to guide changes toward selected areas.
One Union checkpoint supports eight control types:
- Canny
- Depth
- Grayscale
- HED
- Lineart
- MLSD
- Pose
- Scribble
It also supports masked inpainting, where a mask indicates which parts of an image should be regenerated.
For AI artists, this makes it easier to:
- Develop finished artwork from sketches or line drawings.
- Explore different appearances while guiding composition with edges or depth.
- Generate characters following a prepared pose.
- Edit an existing photograph with additional visual references.
- Guide local changes using a painted mask.
The extension uses ComfyUI’s native Qwen Image 2.1 model and conditioning pipeline rather than a separate image-generation system.
Example result using Canny control
The author’s pictured Canny example used the default 40 sampling steps and an output size of 800 × 1312 pixels. This is an example, not a required setting for every image.
Important: The model weights are licensed for non-commercial use only, unless separately licensed by Qwen. See the license section before using them in a project.
How Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 Works
Think of the control map as a drawing guide
A useful analogy is an artist working with a sketch underneath a painting:
- Your prompt describes the desired content and appearance.
- Your control map provides structural guidance.
- The Qwen Image 2.1 base model creates the image.
- The Union ControlNet helps the model follow the supplied structure.
For example, you could use the same prepared line drawing with prompts for a watercolor illustration or a cinematic scene. The drawing guides the arrangement of shapes, while the prompt describes the intended treatment. The result is still generated, so exact correspondence is not guaranteed.
“Union” means several control types share one checkpoint
You do not need a different Union checkpoint for each supported control type. Instead, you select the mode that matches your prepared input.
The mode selector does not turn a photograph into a control map. Selecting Pose, for example, does not detect a person’s pose from an ordinary photograph. You must first prepare the appropriate map using a suitable tool or workflow.
The basic sequence is:
- Prepare a control map.
- Connect it to
control_imageon Apply Qwen Image 2.1 UNION. - Choose the matching control mode.
- Supply your text conditioning and the required model connections.
- Run the workflow to generate an image.
Reference images and control images have different jobs
A control image describes structure. A reference image provides visual information for an edit.
When editing a photograph, use Qwen 2.1 UNION Image Reference Encode. This node prepares the reference information and produces:
- Positive conditioning: information guiding what the model should create.
- Negative conditioning: information used by the workflow to guide against unwanted results.
- Latent: the compressed image representation used during generation.
Connect all three outputs to the sampler as shown in the example workflow.
Always use this encoder’s latent for reference-image editing. Replacing it with an unrelated empty latent can shift the edit or cause the reference guidance to be lost, especially when the aspect ratios differ.
Masks guide local editing, but do not lock pixels
For masked editing, provide the source photograph through inpaint_image and a mask through mask:
- White guides regeneration.
- Black guides preservation.
Think of the mask as an instruction about where to focus the edit, not as an absolute protective barrier. The mask conditions the model and does not guarantee unchanged pixels outside the white area.
Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 Features
Eight structural control modes
Choose the mode according to the kind of visual guidance you have prepared.
| Mode | What the map communicates | Example artistic use |
|---|---|---|
| Canny | Sharp edges and visible boundaries. | Guide the contours of an object or the layout of a scene. |
| Depth | Relative distance and spatial relationships. | Guide foreground/background placement in an interior or landscape. |
| Grayscale | Light and dark tonal information without color. | Use a monochrome image as tonal guidance while describing color in your prompt. |
| HED | Softer detected edges and boundaries. | Guide forms with a soft-edge map rather than a sharp Canny map. |
| Lineart | Defined drawing lines and outlines. | Develop a prepared line drawing into a finished illustration. |
| MLSD | Straight-line structure. | Guide architectural or geometric scenes. |
| Pose | A prepared body-pose representation. | Guide a character’s stance or gesture. |
| Scribble | Loose, simplified drawing marks. | Explore a composition from a rough sketch. |
These modes provide different kinds of guidance; they are not interchangeable labels. A pose map should use Pose, while a prepared depth map should use Depth.
Reference-image editing
The Qwen 2.1 UNION Image Reference Encode node lets you use an existing image as the target of an edit.
The inputs have distinct roles:
| Input | What to provide |
|---|---|
control_image | A prepared map matching the selected control mode. |
reference_image | Image 1: the original image to edit. It determines the output canvas size and aspect ratio. |
reference_image_2 / reference_image_3 | Additional appearance or style references. |
inpaint_image | The source image for masked inpainting. |
mask | A mask using white for regeneration and black for preservation. |
For a two-image edit:
reference_imageis image 1, the edit target.reference_image_2is image 2, an additional visual reference.- Refer to them in your prompt as
<image1>and<image2>.
For example:
Edit
<image1>using the color palette and illustration style of<image2>.
This wording makes the intended roles clearer, but it does not guarantee an exact transfer. In particular, structural control does not guarantee that another person’s face or outfit will be reproduced exactly.
Masked inpainting
Masked inpainting combines a source photograph, a mask, and the workflow’s other guidance.
To prepare a local edit:
- Choose the photograph you want to change.
- Connect it to
inpaint_image. - Create a mask with white over the intended editing area and black elsewhere.
- Connect the mask to
mask. - Describe the desired change in your prompt.
For reference-based masked edits, also follow the reference encoder connections in the supplied workflow. After generation, inspect both the edited region and the surrounding image because preservation is guidance rather than a pixel-level guarantee.
Canvas and aspect-ratio handling
For reference editing, the first reference image determines the canvas. Prepare the control map with the same aspect ratio so its structure does not stretch.
For example, when editing a portrait-oriented photograph, use a portrait-oriented control map rather than a square map containing the same subject.
Extremely wide or tall inputs are rejected if they would produce an output side above 4096 pixels. You can address this by reducing resolution or cropping/padding the input.
Multiple variations from one control
You can explore different seeds while keeping the same structural guide. A seed is the starting randomness used during generation; changing it can produce different interpretations.
However, each connected condition input—control_image, inpaint_image, or mask—must contain one image. The native patch otherwise uses only the first item in each batch.
To sample several seeds from the same control, repeat the latent after the aspect-ratio node, rather than batching multiple condition images.
Native compatibility and checkpoint validation
The extension requires ComfyUI 0.37.0 or newer with native Qwen Image 2.1 base-model support.
It includes Union compatibility for releases before ComfyUI upstream PR #16519. Its loader also validates the expected 16-block Union checkpoint, helping detect an incompatible model file before generation.
Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 Models
The supplied configuration uses one Union checkpoint alongside the Qwen Image 2.1 base-model components. These files perform different jobs; they are not alternative artistic styles.
| Component | Documented file | Role |
|---|---|---|
| Union ControlNet | Qwen-Image-2.1-Fun-Controlnet-Union-ComfyUI.safetensors | Adds the supported structural controls and masked-inpainting guidance. |
| Base diffusion model | qwen_image_2.1_int8_convrot.safetensors | Performs the image-generation process. |
| Text encoder | qwen3vl_8b_int8_convrot.safetensors | Encodes prompt information for the model. |
| VAE | qwen_image_2.1_vae_bf16.safetensors | Converts between visible images and the compressed latent representation used during generation. |
Model resources
- Original Alibaba PAI Union model: upstream model information.
- Converted Union checkpoint for ComfyUI: the checkpoint expected by this extension.
- Comfy-Org Qwen Image 2.1 models: the documented base-model components.
The converted Union checkpoint preserves the original tensor payload bytes; only the safetensors header metadata changes. In simpler terms, the conversion changes the file’s descriptive information rather than the stored model values.
The provided documentation does not list alternative Union sizes or performance comparisons between model variants.
Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 Example Workflows
The workflow folder contains:
- Eight control workflows covering the supported modes.
- A pose-inpainting workflow.
- Reference-image editing.
- Reference-image editing with masked inpainting.
Start with a supplied example
- Make the sample images from
assets/available inComfyUI/input/. - Drag a UI workflow JSON from
workflows/onto the ComfyUI canvas. - Check that the image inputs and model selections resolve correctly.
- Run the example before replacing its inputs.
- Change one element at a time, such as the prompt, control map, or seed.
Files ending in .api.json are API prompts, not workflows intended to be loaded onto the visual canvas.
Starting with an example is especially useful for reference editing because the encoder’s conditioning and latent connections must stay coordinated.
Troubleshooting Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8
The result does not follow the selected control mode
- Inspect the image connected to
control_image. - Confirm that it is already a prepared control map, not an unprocessed photograph.
- Select the mode matching that map.
- Compare your connections with the corresponding example workflow.
Remember: changing the mode selector does not preprocess the image.
A reference edit shifts, loses the source image, or uses the wrong framing
- Confirm that the photograph you want to edit is connected to
reference_image. - Find Qwen 2.1 UNION Image Reference Encode.
- Connect its positive and negative outputs to the sampler.
- Connect its latent output to the sampler.
- Remove any unrelated empty latent being used instead.
- Check that the control map matches the first reference image’s aspect ratio.
The second reference is additional guidance; it does not determine the editing canvas.
The control map looks stretched in the result
- Compare the width-to-height proportions of the target image and control map.
- Crop or pad the map to match the target canvas.
- Check that important structures still align after that adjustment.
- Run the workflow again.
Changes appear in the wrong masked region
- Inspect the mask visually.
- Make the intended editing area white.
- Make the area you want preserved black.
- Confirm that the correct source photograph is connected to
inpaint_image. - Check that the mask lines up with the source photograph.
If changes also appear outside the white area, this is not necessarily a wiring error: the mask does not enforce exact pixel preservation.
Only the first image in a batch affects the result
- Check
control_image,inpaint_image, andmask. - Supply a single image to each connected condition input.
- For multiple variations from the same guidance, repeat the latent after the aspect-ratio node and sample different seeds.
The native patch uses only the first item of a batched condition input.
An extremely wide or tall image is rejected
- Check the intended output dimensions.
- Reduce the input resolution, or crop/pad the image to a less extreme shape.
- Ensure the resulting output would not have a side above 4096 pixels.
- Update the control map to match the revised canvas.
A JSON file does not load as a visual workflow
- Check the filename.
- If it ends in
.api.json, choose the corresponding UI workflow instead. - Drag the UI workflow onto the canvas.
The Union checkpoint fails validation
- Check that you selected the documented converted Union checkpoint.
- Confirm that you have not selected the base diffusion model or another ControlNet by mistake.
- Check the download against the information in
MODEL_SHA256.jsonif file integrity is in doubt. - Verify that ComfyUI meets the documented version and native Qwen Image 2.1 support requirements.
Frequently asked questions
Do I need eight ControlNet checkpoints?
No. One Union checkpoint supports all eight documented control modes.
Can I connect an ordinary photograph and select Canny or Pose?
Not as a substitute for preprocessing. control_image expects a prepared map. Use the reference encoder when the photograph is an editing reference.
Can a second reference determine the output shape?
The first reference_image determines the canvas. Prepare your control map around that target.
Will a pose map copy someone’s face or clothing?
No exact transfer is guaranteed. Pose and other structural controls guide structure, not precise identity or outfit reproduction.
Are 40 sampling steps and 800 × 1312 required?
No. Those are the documented settings for the pictured Canny example, not a universal requirement.
Is this model suitable for commercial artwork?
The supplied model license permits non-commercial use only unless you obtain a separate license from Qwen.
Learn More about Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8
Extension-specific examples
- Example workflows: the most direct starting point for understanding the node connections.
- Sample assets: example images used by the supplied workflows.
- Reference-image editing workflow: demonstrates an edit with reference-image conditioning.
- Reference-image editing and masked inpainting workflow: demonstrates reference guidance combined with a local editing mask.
General ComfyUI learning resources
- ComfyUI workflow library: visual workflow examples for exploring node-based creative work.
- ComfyUI example workflows: additional examples for learning how workflows connect.
- ComfyUI Discord: general help and feedback channels.
- ComfyUI Matrix community (https://app.element.io/#/room/%23comfyui_space%3Amatrix.org): an alternative community space for ComfyUI discussion.
The general ComfyUI resources are useful for learning the interface and troubleshooting workflows, but they are not dedicated documentation for this extension.
Comfyui-Qwen-Image-2.1-Fun-Controlnet-Union-T8 License and Provenance
The extension code and model weights have different licensing terms:
| Material | License |
|---|---|
| Original adapter code | MIT |
| Bundled ComfyUI Union compatibility code | GPL-3.0, adapted from upstream PR #16519 |
| Distributed extension package | GPL-3.0 |
| Model weights | Qwen Research License: non-commercial use only unless separately licensed by Qwen |
For conversion details, see convert_union.py. File checksums are recorded in MODEL_SHA256.json.
The code license does not remove the model’s non-commercial restriction. Review the model license separately when deciding whether the model is appropriate for your intended work.
