logo
RunComfy
AI Toolkit LoRA Training Guides

Qwen 2511 LoRA Training (Qwen-Image-Edit-2511) with Ostris AI Toolkit (Updated Guide)

This tutorial shows how to train Qwen 2511 (Qwen-Image-Edit-2511) LoRAs with Ostris AI Toolkit for multi-image, geometry-aware editing. You'll learn how to build edit datasets (controls + instruction → target), plan VRAM for 1-3 control streams, tune specific parameters, and fix common errors.

Train Diffusion Models with Ostris AI Toolkit

← Scroll horizontally to see full form →

Ostris AI ToolkitOstrisAI-Toolkit

New Training Job

Job

Model

Quantization

Target

Save

Training

Datasets

Dataset 1

Sample

Qwen‑Image‑Edit‑2511 (often shortened to Qwen 2511) is Qwen’s "consistency-first" image editing checkpoint: it’s built to reduce image drift, preserve identity under imaginative edits, and stay structurally faithful when you edit only part of an image. It also ships with integrated LoRA capabilities in the base weights, plus stronger industrial/product design output and improved geometric reasoning, all of which make it especially interesting for practical, repeatable editing LoRAs.

This guide shows how to fine‑tune Qwen 2511 (Qwen‑Image‑Edit‑2511) as a LoRA using Ostris AI Toolkit.

This article is part of the AI Toolkit LoRA training series. If you’re new to Ostris AI Toolkit, start with the AI Toolkit LoRA training overview before diving into this guide.

Table of contents

1. Qwen 2511 vs 2509: what's different

Qwen 2511 is not a "make pretty pictures" checkpoint—it's an instruction-following image editor. If you're coming from Qwen 2509 LoRA Training, think of 2511 as the "consistency-first" iteration: it's tuned to reduce drift, preserve identity/structure, and keep edits localized to what you asked (especially for product/industrial design work and geometry-sensitive placement).

Three differences vs Qwen 2509 matter directly for LoRA training:

First: stronger drift resistance and identity holding. Compared to Qwen 2509, Qwen 2511 tends to keep the "unchanged" parts more stable, which lets your LoRA learn a cleaner edit rule instead of accidentally baking the effect into faces, backgrounds, or composition.

Second: multi-image conditioning is still the core, but the control signal has to be clean. Like Qwen 2509, Qwen 2511 works best when you provide 1–3 reference images plus an instruction. The practical difference is that 2511 rewards well-aligned control streams; if pairing is off or controls are weak, you'll see more over-editing and drift.

Third: more built-in LoRA friendliness (and a stronger need to stay specific). Qwen 2511 ships with stronger integrated LoRA-style capacity in the base weights. That's great for practical, repeatable edit LoRAs, but it also means your LoRA should be trained with a clear, narrow mapping so it doesn’t turn into a vague "everything filter."

2. The core mental model: what an edit LoRA is actually learning

For Qwen 2511, your LoRA is learning a transformation rule:

"Given these reference image(s) and this instruction, produce the edited output while preserving the parts that should remain consistent."

That’s why edit datasets must include all three components:

  • Control/reference image(s): what must be preserved (identity, geometry, lighting, background—whatever your task requires)
  • Instruction (caption/prompt): what must change, stated explicitly
  • Target image: the "after" result that demonstrates the change

If you only provide "after" images, the model has no stable signal for what to keep, so it will learn a noisy shortcut: it may bake changes into identity, background, or composition. That looks like "the LoRA is strong", but it’s actually uncontrolled drift.

The simplest way to judge whether your dataset is "edit-correct" is this: if you remove the instruction, could a human still infer what changed by comparing control(s) to target? If yes, you have a learnable edit signal. If no (or if the change is ambiguous), your LoRA will be fragile.

3. Where to run training: local AI Toolkit vs RunComfy Cloud AI Toolkit

Local AI Toolkit is best when you already have a compatible NVIDIA GPU, you’re comfortable managing CUDA/PyTorch versions, and you want maximum control over files and iteration. (Install AI Toolkit from Ostris’ GitHub repo: ostris/ai-toolkit.) For Qwen 2511, local training can be totally viable—but the model is heavy, and multi-image conditioning can spike VRAM quickly, so you’ll often rely on quantization, low‑VRAM modes, or smaller resolution buckets.

RunComfy Cloud AI Toolkit is the "skip the setup" path and is often the practical choice for Qwen 2511 specifically. You run the same AI Toolkit UI in the browser, but with big GPUs available (and fewer environment surprises). It’s also convenient for teams: datasets, configs, and checkpoints stay in a persistent workspace, so you can iterate like a product workflow instead of a one-off local experiment.

👉 Open it here: Cloud AI Toolkit on RunComfy

4. Hardware & VRAM planning for Qwen‑Image‑Edit‑2511 LoRA

Qwen 2511 is a large backbone and is designed to run at 1024×1024 by default for best results. On top of that, each additional control image stream increases memory use, because the model needs to process more conditioning information.

In practice, you’ll see three workable tiers:

Tier A: 24–32GB VRAM (high-effort, but possible).

Expect to use aggressive strategies: low‑VRAM modes, gradient checkpointing, smaller buckets (often 768 as a starting point), and quantization (ideally with an Accuracy Recovery Adapter option if your build provides it). Keep batch size at 1 and scale with gradient accumulation.

Tier B: 40–48GB VRAM (comfortable).

You can often train at 1024 with one or two control streams, with moderate quantization or even mostly bf16 depending on your exact settings. This tier is where Qwen edit LoRA training becomes "repeatable" rather than "finicky."

Tier C: 80GB+ VRAM (fast, low-friction).

You can keep more components in bf16, run multi-control datasets comfortably, sample more often, and iterate quickly—this is the setup you get with RunComfy Cloud AI Toolkit on big GPUs.

The key idea: resolution and number of control streams are your biggest VRAM levers. If you’re stuck, change those before you start randomly tweaking learning rate.

5. Dataset design that actually works for Qwen edit models

5.1 Folder layout that matches AI Toolkit’s Qwen edit trainer

A practical structure that prevents 90% of bugs:

  • targets/ → the edited "after" images
  • control_1/ → first reference image stream (often the "before" image)
  • control_2/ → second reference stream (optional; second person/product/background/design)
  • control_3/ → third stream (rare; only if your workflow truly needs it)
  • captions/ → optional .txt captions aligned by filename (or captions stored alongside targets depending on your workflow)

The important part is pairing. AI Toolkit can only train correctly if it can match targets/0001.png with control_1/0001.png (and control_2/0001.png, etc.). If file ordering differs, your LoRA learns the wrong mapping and you’ll get "it trains but looks random."

5.2 Three dataset patterns that cover most real LoRAs

Pattern A: Single-reference edit (1 control image).

Use this for: color changes, local object swaps, relighting, background replacement, "turn this into watercolor," etc. Your control_1 is the original image, your target is the edited result, and the caption is a direct instruction ("make the hat red"). This pattern is the easiest to train and debug.

Pattern B: Multi-reference fusion (2–3 control images).

Use this for: person + person, person + scene, product + background, "merge these two identities," or any situation where you want the model to preserve multiple sources. Your captions should clarify the role of each reference ("use person from ref1, background from ref2").

Pattern C: Design insertion triplets (blank + design → applied).

This is the highest ROI dataset pattern for commercial work: logos on shirts, decals on products, patterns on fabric, labels on packaging. control_1 is the product/person without the design, control_2 is the design image, and target is the final "design applied" image. This separation teaches the LoRA exactly what to preserve (geometry/lighting/material) versus what to change (the design region).

5.3 Captions that help (instead of hurting)

For edit LoRAs, your captions should behave like instructions, not descriptions. "A man wearing a shirt, outdoors" is not useful; "Place the provided logo centered on the chest, preserve fabric wrinkles and lighting" is useful.

A good instruction caption usually includes:

  • the intended change
  • what must be preserved
  • any placement or geometry constraints (especially for design insertion)

Keep captions consistent across the dataset. Consistency makes the mapping easier to learn and makes your LoRA more controllable at inference.

5.4 How many samples do you need?

For narrow, repeatable edits (logo insertion, a specific relighting rule, a consistent material transformation), you can often start with 20–60 well-constructed triplets. For broader stylization or multi-subject fusion, plan on 60–200+ examples, because the space of "what should remain consistent" is larger.

If you’re unsure, start small with a "smoke test" set of 8–12 samples. The goal of the smoke test is not quality—it’s to confirm your pairing and controls wiring before you invest in a long run.

6. Step‑by‑step: train a Qwen‑Image‑Edit‑2511 LoRA in AI Toolkit

6.1 Create your datasets in AI Toolkit (Targets + Control Streams)

In DATASETS (see Section 5 for the folder layout logic):

  • Create a dataset for targets/, then add control_1 / control_2 / control_3 if you use them.
  • Verify counts and filename pairing line up across targets and controls (spot-check ~10 samples).
  • If you use captions, set the caption extension (usually .txt) and keep caption filenames matched to targets.

6.2 Create a new job

In JOB:

  • Choose a training name you’ll recognize later.
  • Set a trigger word only if you want the LoRA to be "callable" with a single keyword. For many edit LoRAs, the instruction itself is enough, and a trigger is optional.
  • Set Steps to something conservative for the first run (you’re validating setup, not going for a perfect final model).

In MODEL:

  • Select the Qwen Image Edit "Plus" style architecture (the multi-image edit variant).
  • Set the base model to: Qwen/Qwen-Image-Edit-2511
  • Use bf16 if your GPU supports it; otherwise FP16 can work, but bf16 is usually more stable when available.
  • Enable any "Low VRAM" or offloading options only if you need them; start simple when you can.

In QUANTIZATION (only if you need it):

  • If you’re on 24–32GB, quantize the transformer/backbone first. If your build offers a "with ARA" option for Qwen 2511, prefer that over plain low-bit quantization because it tends to retain more quality.
  • Quantize the text encoder/conditioning side only if VRAM is still tight after transformer quantization.

In TARGET / NETWORK (LoRA settings):

  • Start with a moderate rank. For "rule-like" edits (logo insertion, relighting), you often don’t need extreme rank.
  • If your build exposes separate linear/conv ranks, keep conv conservative unless you have evidence it helps your specific task. Over-parameterizing is a fast path to overfitting and drift.

In TRAINING:

  • Keep Batch Size = 1 and use Gradient Accumulation to increase effective batch if needed.
  • Start with AdamW 8‑bit if you’re VRAM constrained.
  • Use the Qwen-recommended/default scheduler settings your build provides (for Qwen edit jobs this is commonly a flow-matching scheduler).
  • Keep "train text encoder" off for your first successful run unless you have a specific reason to adapt language behavior. Most practical edit LoRAs only need backbone/transformer adaptation.
  • Turn on Gradient Checkpointing if VRAM is tight.

In DATASETS / RESOLUTIONS (Buckets):

  • If you can afford it, 1024 is a strong default for Qwen edit quality.
  • If you’re VRAM constrained, use 768 for the first run, then scale up later once you confirm the pipeline is wired correctly.
  • Prefer a small set of buckets (e.g., 768 and 1024) instead of a chaotic spread that makes the mapping inconsistent.

In SAMPLE / PREVIEWS:

Sampling is your early-warning system. Configure 1–3 preview prompts that represent your real use case, and always use the same fixed control images and seed so you can visually compare checkpoints.

A good sampling cadence for early runs:

  • sample every 100–250 steps early
  • save checkpoints every 250–500 steps
  • keep only a handful of recent checkpoints to avoid disk bloat

6.3 How to know training is working

By ~200–500 steps, you should see at least one of these:

  • the edit begins to happen consistently
  • the preserved parts (identity/background/geometry) stay more stable than "random generation"
  • the change matches the caption instruction directionally

If you only see noise, or the model ignores controls, don’t "fix" it with learning rate first. Fix pairing, controls wiring, and zero_cond_t first.

7. The 2511-specific switch: zero_cond_t

This is an important 2511-specific detail. zero_cond_t changes how timesteps are applied across streams when the model has a denoised stream (the image being generated) and conditioning streams (your reference/control images). With zero_cond_t enabled, the conditioning images are treated as clean references (effectively timestep 0) while the main image follows the normal diffusion timestep schedule.

If your conditioning images are "noised" along with the main stream, the model has a weaker, blurrier reference for identity/structure. That directly increases drift and decreases edit faithfulness. Holding controls at timestep 0 is a clean engineering choice that aligns with the goal of "preserve the reference".

For Qwen 2511, treat zero_cond_t as a compatibility requirement, not a hyperparameter:

  • Enable it for training.
  • Keep it enabled for inference.
  • If your results look unexpectedly driftier than what 2511 is known for, this is the first thing to verify.

8. Common training failures and fixes

8.1 "Missing control images for QwenImageEditPlusModel"

If you see this, AI Toolkit is telling you it did not receive control images at training time. The most common causes are:

  • you attached the targets dataset but didn’t assign control_1 / control_2 in the dataset/job wiring
  • the control folder path is wrong or empty
  • target/control counts don’t match, so controls fail to load for some samples

Fix it by making controls explicit: re-check dataset assignments, confirm folder paths, and ensure filenames/counts match across streams.

8.2 "tuple index out of range" / tensor shape errors early in training

This almost always means the loader expected an image tensor but got None or an unexpected shape. The underlying reasons are usually boring but fixable:

  • a corrupted image file
  • unsupported image mode (CMYK, grayscale)
  • a missing control image for a specific index (pairing mismatch)

Your fix loop should be: validate data integrity → validate pairing → run a tiny smoke test (3–5 samples) before restarting a large job.

8.3 KeyError: 'pixel_values' (often caused by grayscale images)

Qwen edit pipelines typically expect RGB images. Grayscale (single-channel) images can break feature extraction and result in pixel_values errors. Convert your dataset images to standard 3‑channel RGB PNG/JPG and retry.

8.4 Out of memory (OOM), especially during sampling

Multi-image edit training can spike VRAM during preview sampling because it runs additional forward passes and may use larger intermediate buffers.

Fix OOM in this order:

  1. reduce preview frequency or preview resolution
  2. keep batch size at 1, increase gradient accumulation
  3. reduce buckets (or drop to 768)
  4. enable quantization/offloading
  5. temporarily train with fewer control streams while debugging
  6. if you’re still OOM locally, run the same job in RunComfy Cloud AI Toolkit on a larger GPU

8.5 LoRA loads but "does nothing" (or loads with missing keys) in ComfyUI

When a LoRA does nothing, it’s usually one of:

  • you’re loading it into a different architecture than it was trained for
  • the LoRA scale is too low to notice
  • there’s a key-prefix mismatch between what the inference stack expects and what the trainer saved

If you see missing key warnings specifically for Qwen LoRAs, one known workaround is to rewrite the LoRA state dict key prefix (for example, mapping diffusion_model. keys to transformer. keys). If your AI Toolkit build and your ComfyUI nodes are both updated, this may already be fixed—but it’s the first thing to try when you see systematic "keys not loaded" issues.

9. Using your trained LoRA (Playground + ComfyUI)

Once training is complete, the fastest way to sanity-check your Qwen 2511 LoRA is to load it in the Qwen‑Image‑Edit‑2511 LoRA Playground; when you want a repeatable node graph for real work, start from the Qwen‑Image‑Edit‑2511 ComfyUI workflow and swap in your LoRA.

More AI Toolkit LoRA training guides

Ready to start training?