kanibus Introduction
kanibus is a collection of ComfyUI nodes for tracking eyes in images and video and turning visual information into controls for an AI image or video workflow. It is designed to help you guide details that can be difficult to keep consistent across frames, such as where a character looks, when they blink, and how their face and body move.
The extension combines eye tracking with other visual guides, including eye masks, depth maps, surface-normal maps, facial landmarks, hand tracking, and body pose. You can use only the parts your project needs or bring several together in one workflow. For example, an eye mask can help focus control on the eyes, while a pose guide can help preserve a character’s movement.
kanibus is described as supporting workflows involving WAN 2.1 and WAN 2.2. Its listed ControlNet files, however, are SD 1.5 models. Check that the models and connections in your chosen ComfyUI workflow are compatible with the image or video model you intend to use; the readme does not establish that those ControlNet files can be connected directly to every WAN setup.
How kanibus Works
Think of kanibus as a set of guides prepared from a reference image or video:
- Read the source. A video loader separates a clip into frames, or you provide an image or webcam input.
- Find useful visual details. Tracking nodes identify features such as eyes, facial points, hands, and body position. The eye tracker can estimate pupil position, gaze direction, and blinks.
- Make control images. Other nodes turn detected details into guides such as an eye mask, depth map, or pose visualization. A depth map represents which areas appear nearer or farther away; a normal map describes the apparent direction of surfaces.
- Apply and smooth the guides. ControlNet conditioning can use the guides to influence generation. Temporal smoothing aims to reduce abrupt changes between video frames.
For a simple example, you could load a reference clip, track the subject’s eyes, and use the resulting eye mask to guide an output clip. A more involved workflow could add depth and pose guides so the character’s gaze, position, and movement follow the reference more closely.
Tracking and generation are separate jobs: detecting a gaze direction does not, by itself, guarantee that a generated character will look exactly the same way. The result also depends on your model, workflow connections, source quality, and control settings.
kanibus Features
Eye tracking and expression cues. The Neural Pupil Tracker uses facial and iris landmarks to follow the eyes. It provides tracking data, an annotated image, a gaze visualization, and separate left- and right-eye masks. It also includes blink detection, rapid-eye-movement detection, and pupil-size tracking. These are visual measurements, not a reliable way to determine someone’s real emotions.
You can adjust sensitivity when detection misses the eyes or responds too readily to surrounding details. smoothing can make motion steadier, although too much smoothing may make quick glances feel less responsive. blink_threshold changes when an eye closure counts as a blink, while saccade_threshold affects detection of fast gaze shifts.
Video loading. The Video Frame Loader accepts common formats listed by the author, including MP4, AVI, MOV, MKV, and WEBM. It offers quality and frame-processing options and uses caching to avoid repeating some work. A lower resolution or fewer processed frames can make a long clip easier to work with, at the cost of fine detail or motion coverage.
Combined processing. Kanibus Master brings several controls into one node. You can select an input source, choose a processing mode, enable the features you need, and set options such as target frame rate, tracking quality, temporal smoothing, and individual control weights. Its outputs include processed imagery, control maps, conditioning, and a processing report. Start with a small number of enabled controls when you want to understand which one is changing your result.
Specialized guides. The node list also includes Smart Facial Masking, AI Depth Control, Normal Map Generator, Landmark Pro 468, Hand Tracking, Body Pose Estimator, and Object Segmentation. These offer ways to isolate a face or object or describe its shape and position. Advanced Tracking Pro is listed for multi-object tracking, and Emotion Analyzer for expression-related analysis. The original documentation does not give detailed settings for these nodes, so inspect their inputs and outputs in ComfyUI before relying on a particular behavior.
Frame consistency and ControlNet. Temporal Smoother is intended to make guides change more steadily across frames. Multi-ControlNet Apply combines control outputs for generation. The documented controls have separate weights: increasing a weight generally asks the workflow to follow that guide more strongly, while lowering it gives the generation more freedom. If a face looks rigid, try reducing one control at a time; if the eyes drift, try increasing the eye control gradually.
Processing options. The author describes GPU acceleration, mixed-precision options, caching, and batch processing. Actual speed and memory use will vary with hardware, resolution, enabled nodes, and the generation workflow. Treat the frame rates and benchmark figures in the original readme as reported targets, not guaranteed results.
kanibus Models
The original readme identifies four required ControlNet model files. They serve different kinds of visual guidance:
| Model file | Guide | When it may help |
|---|---|---|
control_v11p_sd15_scribble.pth | Eye-mask control | Emphasizing the shape or location of the eye area |
control_v11f1p_sd15_depth.pth | Depth-map control | Preserving the apparent near-to-far layout of a scene |
control_v11p_sd15_normalbae.pth | Normal-map control | Guiding the apparent orientation and form of surfaces |
control_v11p_sd15_openpose.pth | Pose control | Following a person’s body position or movement |
These are four different control types, not four interchangeable versions of the same model. For example, choose depth when spatial layout matters and pose when a character’s stance matters. Combining guides may offer more direction, but it can also constrain the result too much.
The readme also names WAN 2.1 and WAN 2.2 as supported video-workflow versions and describes an auto_detect option in Kanibus Master. These are not the four ControlNet files above. The example workflows are a useful starting point for seeing how the author intended the pieces to connect:
examples/wan21_basic_tracking.json— a basic WAN 2.1 eye-tracking workflow.examples/wan22_advanced_full.json— a fuller WAN 2.2 workflow.examples/realtime_webcam.json— a webcam-oriented workflow.
For the author’s model-file details, see REQUIRED_MODELS.md.
Troubleshooting kanibus
The Kanibus nodes do not appear in ComfyUI
- Fully close and restart ComfyUI.
- Search the node menu for the Kanibus category. The original readme lists 14 nodes.
- Check ComfyUI’s startup messages for an error related to Kanibus or a missing dependency.
- If the category still does not appear, use the project’s GitHub Issues to compare your error message with reported problems.
A workflow says a ControlNet model is missing
- Read the missing filename in the error message.
- Compare it with the four files in REQUIRED_MODELS.md.
- Confirm that the file is present in ComfyUI’s
models/controlnetfolder and that its name matches the workflow’s selection. - Reopen or refresh the relevant model selector in ComfyUI, then try the workflow again.
Eye tracking misses the eyes or looks unstable
- Check the source frame. Eyes that are tiny, hidden, heavily blurred, or out of view give the tracker less to work with.
- View the tracker’s annotated image and gaze visualization to see what it detected before judging the generated result.
- Adjust
sensitivityin small steps if the eyes are missed or nearby details are mistaken for eyes. - Increase
smoothinggradually if the detected position jumps between frames. Reduce it if fast eye movements seem delayed.
The generated video flickers or the control looks too strong
- Preview the individual masks, maps, or pose outputs to identify which guide changes abruptly.
- Try Temporal Smoother for frame-to-frame changes.
- Lower one ControlNet weight at a time if the output looks rigid or distorted.
- Test a short section of the clip after each change so you can tell which setting helped.
Processing is slow or runs out of memory
- Try a shorter clip or lower-resolution input.
- Disable guides you do not need and reduce batch size if your workflow exposes that setting.
- Close other applications using substantial GPU memory.
- Check the processing report, if your workflow provides one, to see whether a particular stage is taking most of the time.
Frequently asked questions
Do I need every guide for eye tracking? No. A basic workflow can focus on video loading and eye tracking. Add depth, pose, or other guides when they solve a specific problem in your result.
Does kanibus generate a finished video on its own? The documented nodes primarily load, analyze, and prepare controls for a connected generation workflow. Your output also depends on the generation nodes and models you use in ComfyUI.
Will a higher control weight always improve accuracy? Not necessarily. Stronger guidance may follow a reference more closely, but it can also restrict expression or introduce unwanted artifacts. Adjust weights gradually while reviewing the output.
Are the reported frame rates guaranteed? No. The readme provides performance figures, but speed depends on your computer and the work each frame requires.
Learn More about kanibus
- Project documentation — additional documentation referenced by the author.
- Example workflows — starting points for basic tracking, a fuller pipeline, and webcam processing.
- Required model guide — filenames and further details for the four listed ControlNet files.
- GitHub Discussions — a place to ask workflow questions and compare approaches.
- GitHub Issues — reported problems and a place to provide a reproducible error when something does not work.
The original readme includes a roadmap for possible future features, but it does not provide a verified version-by-version record of released changes. Planned features should not be treated as available until the project documents their release.
