ComfyUI Node: IF Video Prompts 🎥🧠

IF Video Prompts 🎥🧠:

The VideoPromptNode is a versatile tool within the ComfyUI framework designed to analyze video sequences or files using advanced Qwen2.5-VL multimodal models. This node offers two primary modes of operation: frame-based mode, which processes pre-loaded frames from LoadVideo nodes, and direct video mode, which handles video files directly. Its capabilities include generating descriptive prompts for video content, conducting comprehensive scene analyses, breaking down key scenes, and creating negative prompts based on configured templates. By leveraging these features, the node provides AI artists with insightful and detailed textual descriptions of video content, enhancing their ability to understand and creatively engage with video data.

IF Video Prompts 🎥🧠 Input Parameters:

profile

This parameter allows you to select a predefined profile that includes a system prompt and specific rules for video analysis. The default profile is "HyVideoAnalyzer

• Simple one line prompt," which provides a straightforward analysis approach. Profiles help tailor the analysis to specific needs or styles.

max_new_tokens

This integer parameter sets the maximum number of new tokens the model can generate during analysis. It ranges from 1 to 2048, with a default value of 512. Adjusting this value can impact the length and detail of the generated prompts.

frame_sample_count

This integer parameter determines the number of frames sampled from the video sequence for analysis. It ranges from 1 to 32, with a default of 16. Increasing the count can provide a more comprehensive analysis but may require more processing power.

temperature

A float parameter that controls the randomness of the generated text. It ranges from 0.0 to 2.0, with a default of 0.7. Higher values increase creativity but may reduce coherence, while lower values produce more deterministic outputs.

analysis_type

This parameter specifies the type of analysis to perform, with options including "Full sequence," "Key scenes," and "Single summary." The default is "Full sequence," which provides a detailed analysis of the entire video.

language

This parameter sets the language for the output, with options for "English" and "Chinese." The default is "English," allowing you to receive prompts in your preferred language.

images

An optional parameter that accepts input frames from a LoadVideo node. These frames are used in frame-based mode for analysis.

video_file

An optional parameter that allows you to select a video file from the input directory for direct video mode processing.

fps

This float parameter sets the frames per second for video processing, ranging from 0.1 to 60.0, with a default of 8.0. Higher values sample more frames, potentially improving analysis quality.

max_pixels

An integer parameter that defines the maximum number of pixels for video processing, with a default of 512512. It ranges from 0 to 1280720, and setting it to 0 uses the default resolution.

fallback_frame_count

This integer parameter specifies the number of frames to use in fallback mode if initial processing fails. It ranges from 1 to 16, with a default of 4. Lower values use less VRAM, which can be beneficial for resource-constrained environments.

custom_system_prompt

An optional string parameter that allows you to provide a custom system prompt to override the selected profile. This can be useful for tailoring the analysis to specific requirements.

prefix

An optional string parameter that adds text before the generated prompt, allowing for customization of the output.

suffix

An optional string parameter that adds text after the generated prompt, providing additional customization options.

seed

An integer parameter for setting a random seed for generation, with a default of -1 for random. This can be used to ensure reproducibility of results.

negative_prompt

This parameter allows you to select a predefined negative prompt to use, which can help refine the analysis by excluding certain elements.

model_offload

A parameter with options "Yes" or "No" that determines whether to offload the model from the GPU when not in use, saving VRAM. The default is "Yes."

precision

This parameter sets the model precision, with options "float16," "bfloat16," and "float32." The default is "float16," balancing performance and resource usage.

IF Video Prompts 🎥🧠 Output Parameters:

generated_text

The primary output of the node is a generated text prompt that captures the essence of the analyzed video content. This text can include descriptive prompts, scene analyses, and key scene breakdowns, providing valuable insights into the video.

IF Video Prompts 🎥🧠 Usage Tips:

• To achieve a more detailed analysis, increase the frame_sample_count to capture more frames from the video sequence.
• Use the temperature parameter to adjust the creativity of the generated text. A higher temperature can lead to more imaginative descriptions, while a lower temperature ensures more consistent outputs.
• If you have specific analysis requirements, consider using the custom_system_prompt to tailor the system prompt to your needs.
• For resource-constrained environments, set model_offload to "Yes" to save VRAM when the model is not in use.

IF Video Prompts 🎥🧠 Common Errors and Solutions:

Failed to analyze video: <error_message>

• Explanation: This error occurs when the node encounters an issue during video analysis, possibly due to incorrect input parameters or unsupported video formats.
• Solution: Verify that the input video file or frames are correctly formatted and supported by the node. Check the input parameters for any inconsistencies or errors.

Model loading error

• Explanation: This error indicates a problem with loading the Qwen2.5-VL model, which may be due to insufficient VRAM or incorrect model settings.
• Solution: Ensure that your system has enough VRAM to load the model. Consider adjusting the precision parameter to a lower setting, such as "float16," to reduce VRAM usage.