ComfyUI  >  Workflows  >  AnimateDiff + ControlNet | Ceramic Art Style

AnimateDiff + ControlNet | Ceramic Art Style

This workflow within ComfyUI employs AnimateDiff and ControlNet featuring on depth, among others like Lora, to adeptly transform videos into Ceramic Art Style. It enables the original content to adopt a distinct, artistic flair, effectively elevating it to the realm of ceramic art masterpieces.

ComfyUI Vid2Vid (Art) Workflow

Transform Video to Ceramic Art Style with AnimateDiff and ControlNet within ComfyUI
Want to run this workflow?
  • Fully operational workflows
  • No missing nodes or models
  • No manual setups required
  • Features stunning visuals

ComfyUI Vid2Vid (Art) Examples

ComfyUI Vid2Vid (Art) Description

1. ComfyUI Workflow: AnimateDiff + ControlNet | Ceramic Art Style

This workflow utilizes AnimateDiff, ControlNet focusing on depth, and specific Lora to skillfully transform videos into a Ceramic Art Style. You are encouraged to use different prompts to achieve various art styles, turning your ideas into reality.

2. How to Use AnimateDiff

AnimateDiff is designed to animate static images and text prompts into dynamic videos, leveraging Stable Diffusion models and a specialized motion module. It automates the animation process by predicting seamless transitions between frames, making it accessible to users without coding skills .

2.1 AnimateDiff Motion Modules

To begin, select the desired AnimateDiff motion module from the model_name dropdown:

  • Use v3_sd15_mm.ckpt for AnimateDiff V3
  • Use mm_sd_v15_v2.ckpt for AnimateDiff V2
  • Use mm_sdxl_v10_beta.ckpt for AnimateDiff SDXL
AnimateDiff Motion Modules in ComfyUI

2.2 Beta Schedule

The Beta Schedule in AnimateDiff is crucial for adjusting the noise reduction process throughout animation creation.

For versions V3 and V2 of AnimateDiff, the sqrt_linear setting is recommended, although experimenting with the linear setting can yield unique effects.

For AnimateDiff SDXL, the linear setting (AnimateDiff-SDXL) is advised.

AnimateDiff Beta Schedule in ComfyUI

2.3 Motion Scale

The Motion Scale feature in AnimateDiff allows for the adjustment of motion intensity in your animations. A Motion Scale under 1 results in more subtle motion, whereas a scale over 1 amplifies movement.

AnimateDiff Motion Scale in ComfyUI

2.4 Context Length

The Uniform Context Length in AnimateDiff is essential for ensuring seamless transitions between scenes defined by your Batch Size. It acts like an expert editor, seamlessly connecting scenes for fluid narration. Setting a longer Uniform Context Length ensures smoother transitions, while a shorter length offers quicker, more distinct scene changes, beneficial for certain effects. The standard Uniform Context length is set to 16.

AnimateDiff Context Length in ComfyUI

2.5 Utilizing Motion LoRA for Enhanced Camera Dynamics (Specific to AnimateDiff v2)

Motion LoRAs, compatible solely with AnimateDiff v2, introduce an additional layer of dynamic camera movement. Achieving the optimal balance with the LoRA weight, typically around 0.75, ensures smooth camera motion free from background distortions.

Moreover, chaining various Motion LoRA models allows for complex camera dynamics. This enables creators to experiment and discover the ideal combination for their animation, elevating it to a cinematic level.

AnimateDiff Motion LoRA in ComfyUI

3. How to Use ControlNet

ControlNet enhances image generation by introducing precise spatial control to text-to-image models, allowing users to manipulate images in sophisticated ways beyond just text prompts, by utilizing vast libraries from models like Stable Diffusion for intricate tasks such as sketching, mapping, and segmenting visuals.

The following is the simplest workflow using ControlNet.

3.1 Loading the "Apply ControlNet" Node

Start your image crafting by loading the "Apply ControlNet" Node in ComfyUI, setting the stage for combining visual and textual elements in your design.

3.2 Inputs of "Apply ControlNet" Node

Use Positive and Negative Conditioning to shape your image, select a ControlNet model to define style traits, and preprocess your image to ensure it matches the ControlNet model requirements, thereby making it ready for transformation.

3.3 Outputs of "Apply ControlNet" Node

The node outputs guide the diffusion model, offering a choice between refining the image further or adding more ControlNets for enhanced detail and customization based on the interaction of ControlNet with your creative inputs.

3.4 Tuning "Apply ControlNet" for Best Results

Control the influence of ControlNet on your image through settings like Determining Strength, Adjusting Start Percent, and Setting End Percent to finely tune the image's creative process and outcome.

ControlNet in ComfyUI

For more detailed information, please check out How to use ControlNet in ComfyUI

This workflow is inspired by MDMZ with some modifications. For more information, please visit his YouTube channel.

More ComfyUI Tutorials

Want More ComfyUI Workflows?


© Copyright 2024 RunComfy. All Rights Reserved.

RunComfy is the premier ComfyUI platform, offering ComfyUI online environment and services, along with ComfyUI workflows featuring stunning visuals.