AnimateDiff + Batch Prompt Schedule | Text to Video

1. AnimateDiff + Batch Prompt Schedule Workflow

The AnimateDiff and Batch Prompt Schedule workflow enables the dynamic creation of videos from textual prompts. By allowing scheduled, dynamic changes to prompts over time, the Batch Prompt Schedule enhances this process, offering intricate control over the narrative and visuals of the animation and expanding creative possibilities for storytelling.

2. Overview of AnimateDiff

Please check out the details on How to use AnimateDiff in ComfyUI

3. Overview of Batch Prompt Schedule

3.1. Introduction to Batch Prompt Schedule

The Batch Prompt Schedule in ComfyUI is a powerful node designed for efficiently managing and scheduling complex prompts across a series of frames or iterations. It enables creators to dynamically adjust text and parameters over time, allowing for detailed control in animation and other time-based media projects.

3.2 How to Use Batch Prompt Schedule

Firstly, it's important to note the three text fields for prompts in the Batch Prompt Schedule node: Initial Text Field, Pretext Field, and Posttext Field.

To compose an Initial Text Field in a ComfyUI Batch Prompt Schedule, you essentially define the core or base prompt that will be applied across all frames or iterations in your schedule. This is accomplished using the Pretext Field or Posttext Field, which respectively allows you to prepend or append text to every scheduled prompt.

Initial Text Field: This is the main text input where you enter the prompts you want processed by the Batch Prompt Schedule. Specify the prompt for keyframes here. For example:

"0": "Grim Reaper, standing among the graves, carrying a scythe, in a misty graveyard at night, eerie fog, full moon, bats flying around, Halloween pumpkins",
"25": "Male zombie, open mouth, eating a brain, in an abandoned post-apocalyptic cityscape, graffiti walls, flickering streetlights",
"50": "Werewolf howling, his eyes glowing red, in a dense eerie forest, full bright moon, pack of wolves in the background"

Pretext Field: Text that is prepended at the beginning of each prompt in the schedule, allowing for a consistent base across all scheduled prompts.

Posttext Field: Text that is appended at the end of each prompt in the schedule, enabling a uniform conclusion to each prompt or adding consistent elements across prompts.

Additionally, there are some other parameters to help control the prompts:

Max Frames: Determines the maximum number of frames for which the batch prompt schedule is applied. It is crucial to align this number with the number of latents in your batch or the total animation frames you intend to produce.

Print Output: A boolean parameter that, when enabled, allows for the output to be printed for debugging or verification purposes.

Start Frame: Specifies the starting frame for the batch prompt schedule.

PW Fields: These fields (pw_a, pw_b, pw_c, pw_d) facilitate the dynamic adjustment of prompt weights over time. By manipulating these weights through expressions and linking them to other scheduled values, creators gain nuanced control over how different aspects of the prompt influence the batch schedule's output. For instance, you can incorporate pw_a, pw_b, pw_c, and pw_d into the Initial Text Field as follows:

"0": "A joyful dancer in the spotlight, smiling broadly (happiness:`pw_a`)",
"24": "A contemplative dancer under the moonlight, lost in thought (contemplation:`pw_b`)",
"48": "A sorrowful figure in the shadows, tears glistening in the moonlight (sorrow:`pw_c`)"

This workflow is inspired by MDMZ. For more information, please visit his YouTube channel.