VideoX-Fun Introduction
VideoX-Fun is a toolkit for creating AI images and videos. You can generate a clip from a written description, animate a still image, transform a reference video, or use visual guides to influence motion and composition. It also supports image generation and editing through compatible image models.
The project brings many model families into a common workflow. You can use existing models to make artwork, or train a base model or a LoRA to adapt a model to a particular visual style. A LoRA is a smaller add-on used with a compatible base model; it does not replace the base model.
VideoX-Fun offers several ways to work:
- Web interface: An interactive option for supported video-model families.
- ComfyUI: A node-based option for artists who already build ComfyUI workflows.
- Python scripts: An option for batch work and settings not exposed in the web interface.
You can explore the hosted CogVideoX-Fun demo and Wan-Fun demo before choosing a local workflow.
How VideoX-Fun Works
Think of a model as an artist that has learned visual patterns, and your inputs as its brief. VideoX-Fun passes those inputs to the model you select and saves the resulting image or video.
Different tasks provide different kinds of guidance:
- Text to video: You describe a scene and the model creates a moving clip. For example, you might request a person walking along a rainy street.
- Image to video: You provide a still image as a starting point and describe what should move. Some supported models also accept an ending image, giving the clip a destination as well as a starting point.
- Video to video: You provide a reference clip whose content or motion guides a new result.
- Controlled video generation: You provide a processed guide, such as an edge map, depth map, or body-pose sequence. This is like giving the model a rough storyboard for structure or movement.
- Text to image and image editing: Compatible image models create an image from a description or edit an existing one.
The prompt describes what you want; a negative prompt describes elements you would rather avoid. A seed controls the random starting point: keeping it the same can help you compare changes to other settings. Guidance scale adjusts how strongly generation follows the prompt. Video duration, frame rate (FPS), and resolution are available according to the chosen model and workflow. More frames or a larger image generally make a generation task more demanding, so begin with a modest test before committing to a larger output.
VideoX-Fun Features
Choose the workflow that fits your project
The web interface supports text-to-video, image-to-video, video-to-video, and ordinary control-guided video generation for the CogVideoX-Fun, Wan2.1, Wan2.1-Fun, Wan2.2, and Wan2.2-Fun families. You can select a model, enter prompts and other common settings, and generate a result. Other models, including the image models described in the source README, use Python prediction scripts rather than this web interface.
If you prefer arranging steps visually, see the ComfyUI guide. This example image-to-video workflow (https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/cogvideoxfunv1_workflow_i2v.jpg) shows what a node graph can look like.
Python scripts provide the broadest control. Their names indicate the task: predict_t2v.py for text-to-video, predict_i2v.py for image-to-video, predict_v2v.py for video-to-video, and, where available, predict_t2i.py for text-to-image. Check the scripts in examples/{model_name}/ to see which tasks a particular model actually supports.
Guide the result with images and video
For image-to-video work, validation_image_start supplies the opening image and validation_image_end can supply the ending image. This is useful when you need a clip to begin and finish on particular compositions.
For video-to-video work, validation_video supplies a reference clip. For controlled generation, control_video supplies a prepared guide such as Canny edges, Pose, or Depth. You can inspect the source README’s reference video (https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/play_guitar.mp4) and pose-control example (https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1.1/pose.mp4) to understand the difference between a clip and a processed motion guide.
Balance GPU memory and speed
Large models can require substantial graphics-card memory. VideoX-Fun provides several memory modes so you can trade speed for a better chance of fitting a model on your GPU:
| Mode | What it means for you |
|---|---|
model_full_load | Keeps the model on the GPU. Fastest option, but uses the most GPU memory. |
model_cpu_offload | Moves the model to the CPU after use, reducing GPU-memory needs. The source README recommends this when you have enough memory to use it. |
model_cpu_offload_and_qfloat8 | Combines CPU offloading with lower-precision transformer weights to save more memory; lower precision can reduce model performance. |
sequential_cpu_offload | Moves individual layers to the CPU after use. Saves substantial GPU memory but is slower. |
model_group_offload | Moves groups of weights between CPU and GPU, aiming to balance memory use and speed. Available in prediction scripts. |
model_full_load_and_qfloat8 | Keeps the model on the GPU while reducing transformer precision. Available in prediction scripts. |
The web interface exposes only model_full_load, model_cpu_offload, model_cpu_offload_and_qfloat8, and sequential_cpu_offload. The other two modes require a Python prediction script. The source README also notes that compile-based acceleration is incompatible with sequential_cpu_offload and fsdp_dit.
Adapt a model to your style
VideoX-Fun supports training both base models and LoRAs. Training data pairs images or videos with text descriptions. For an artist, a LoRA can be a way to explore a consistent visual treatment while continuing to use a compatible base model. Training is a more advanced workflow than generating with an existing model; consult the model-specific guides before preparing a dataset.
VideoX-Fun Models
Choose by task first, then by model family. A name containing T2V indicates text-to-video, I2V image-to-video, and TI2V support for both. InP models in the supplied catalog cover text/image-guided generation, including start-and-end-image use where noted. Control models accept additional structural guidance. A larger number such as 14B identifies a larger model, not a promised quality or speed improvement on your hardware.
The following are the model weights fully identifiable in the supplied context. Links lead to their Hugging Face pages.
| Family | Models | When to consider them |
|---|---|---|
| Wan2.2-Fun | A14B-InP, 5B-InP | Text- or image-guided video, including start-and-end images. The listed A14B model is described as 81 frames at 16 FPS; the 5B model as 121 frames at 24 FPS. |
| Wan2.2-Fun Control | A14B-Control, 5B-Control | Video guided by Canny edges, depth, pose, MLSD lines, or trajectories. |
| Wan2.2-Fun camera control | A14B-Control-Camera, 5B-Control-Camera | Projects where you want to guide camera movement as well as the generated scene. |
| Wan2.2-VACE-Fun | A14B | Control-guided video or subject-reference generation. |
| Wan2.2 official models | TI2V-5B, T2V-A14B, I2V-A14B, S2V-14B, Animate-14B | Respectively: combined text/image-to-video, text-to-video, image-to-video, speech-driven digital humans, and character replacement or motion transfer. |
| Wan2.1-Fun V1.1 InP | 1.3B-InP, 14B-InP | Text- and image-guided video with start-and-end-image support. The supplied catalog describes this series as supporting multiple resolutions and 81 frames at 16 FPS. |
| Wan2.1-Fun V1.1 Control | 1.3B-Control, 14B-Control | Control-guided video, including a reference image combined with control guidance. |
| Wan2.1-Fun V1.1 camera control | 1.3B-Control-Camera, 14B-Control-Camera | Camera-motion guidance in the V1.1 series. |
| Wan2.1-Fun V1.0 | 1.3B-InP, 14B-InP, 1.3B-Control, 14B-Control | Earlier InP and control options; the supplied catalog says V1.0 does not include the V1.1 camera-control capability. |
| Wan2.1 official models | T2V-1.3B, T2V-14B, I2V-14B-480P, I2V-14B-720P | Task-specific original Wan2.1 weights. Choose a T2V model for a prompt alone or an I2V model to animate an image. |
The catalog also lists Wan2.2-Fun Reward LoRAs and Wan2.1-Fun Reward LoRAs. These are add-ons for compatible weights, not standalone generation models. VideoX-Fun additionally names CogVideoX-Fun, HunyuanVideo, MiniMax-H3, LTX-2, LongCat-Video, FantasyTalking, LingBot, Qwen-Image, Z-Image, Flux/Flux2, and ERNIE-Image as supported families, but the provided context does not include a complete, usable weight list for them.
What's New with VideoX-Fun
The supplied context does not provide a dated release history. It does identify these model-series differences:
- Wan2.1-Fun V1.1 compared with V1.0: V1.1 includes camera-control models. That matters when a shot needs a guided camera move rather than only subject or scene control.
- Wan2.2-Fun options: The catalog includes InP, control, and camera-control weights in both A14B and 5B variants, giving you separate choices for open-ended generation and more directed work.
- Wan2.2-VACE-Fun: Adds a listed route for control-guided or subject-reference video generation.
These are descriptions of available model families, not a claim about when each change was released.
Troubleshooting VideoX-Fun
The model does not appear or cannot be loaded
- Confirm that you obtained the weight for the exact model name you selected.
- For the project's scripts or web interface, check that its folder is under
models/Diffusion_Transformer/and that the folder name matches the weight name. - For ComfyUI, check
ComfyUI/models/Fun_Models/instead. - If you are loading a trained model or LoRA in the web interface, check
models/Personalized_Model/. - Consult the model's page to confirm you have the files required by that weight.
The task I want is missing from the web interface
- Check whether your model family is one of the web-interface families listed above.
- Look in
examples/{model_name}/for a script matching the task, such aspredict_t2i.pyorpredict_i2v.py. - If no matching script exists, do not assume that the model supports that task. Choose a listed model designed for it.
Generation runs out of GPU memory
- Try a smaller model or a less demanding output setting.
- In the web interface, switch from full loading to
model_cpu_offload. - If memory is still insufficient, consider
model_cpu_offload_and_qfloat8orsequential_cpu_offload, accepting a potential quality or speed trade-off. - If you use a Python prediction script, review its additional
GPU_memory_modechoices.
The result does not follow my intended motion or composition
- Check that you chose the right task: a starting image, reference video, and processed control video serve different purposes.
- For image-to-video, verify that the opening image is assigned to
validation_image_start; usevalidation_image_endwhen your selected workflow supports an ending image. - For controlled generation, make sure
control_videocontains the intended processed guide, such as pose or depth, rather than assuming an ordinary clip will act as one. - Revise the prompt to describe the action clearly, then compare results while keeping the seed fixed.
Frequently asked questions
Do I need to train a model before making art?
No. VideoX-Fun supports generating with already-trained models. Training is an optional path for adapting a model.
Can I use the same interface for every image and video model?
No. The source README lists web interfaces for specific video families; other models, including image models, use prediction scripts. ComfyUI is another option where a compatible workflow is available.
What is the difference between a base model and a LoRA?
A base model performs the main generation. A LoRA is a smaller adaptation used alongside a compatible base model, often for a particular style or behavior.
Will a memory-saving mode make generation faster?
Not necessarily. Moving model parts between CPU and GPU can reduce GPU-memory use but slow generation. Lower-precision modes can also affect model performance.
Learn More about VideoX-Fun
- Try a hosted example: CogVideoX-Fun on Hugging Face Spaces or Wan-Fun on Hugging Face Spaces.
- Build a visual workflow: Read the ComfyUI guide and examine the image-to-video workflow image (https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/cogvideoxfunv1_workflow_i2v.jpg).
- Explore style adaptation: Start with the Wan2.1-Fun LoRA training guide, Wan2.2-Fun LoRA training guide, CogVideoX-Fun LoRA training guide, or Qwen-Image LoRA training guide, depending on the model you use.
- Prepare described video material: See the video-captioning guide. This linked guide is in Chinese.
