ComfyUI Node: OmniCam Extractor

OmniCam Extractor:

The MajoorOmniCamExtractor is a powerful node designed to extract a relative 6 Degrees of Freedom (6DoF) camera trajectory from a continuous video shot, transforming it into a canonical OmniCam motion scene. This node is particularly beneficial for AI artists and creators who need to analyze and replicate camera movements within digital environments. By leveraging advanced visual odometry techniques, the node can detect and interpret complex camera motions such as translation, rotation, and orbital arcs, as well as optical dynamics like dolly zooms. This capability allows for the creation of realistic and dynamic camera paths that can be used in various applications, from virtual reality to cinematic productions. The node is experimental, indicating that it is at the forefront of technology, offering cutting-edge features that are continuously being refined and improved.

OmniCam Extractor Input Parameters:

video

This parameter accepts a continuous video shot or an image batch, which serves as the source material for extracting the camera trajectory. The video should not contain hard cuts, as these are reported and not stitched. The quality and continuity of the video directly impact the accuracy of the camera motion extraction.

extract_mode

This parameter determines the mode of operation for the node. It offers two options: camera_track and scene_reconstruct. The camera_track mode solves the 6DoF camera motion from the video, while scene_reconstruct recovers a 3D proxy scene from a single still image. The default mode is camera_track.

method

This parameter specifies the method used for solving the camera motion. Options include auto, dpvo, pycolmap, and opencv_sift. The auto option selects the first available solver, prioritizing DPVO, then pycolmap, and finally OpenCV/SIFT. Users can choose a specific solver to force its use and receive installation hints if it is missing. The default is auto.

lens_mode

This parameter defines the lens mode used during extraction. Options are auto, fov, and focal_mm. The auto mode automatically selects the appropriate lens mode based on the input video. This parameter is advanced and typically used by those with specific lens requirements.

fov_degrees

This parameter sets the field of view in degrees, with a default value of 53.0. It can range from 10.0 to 140.0 degrees, allowing users to adjust the field of view to match the characteristics of the input video. This is an advanced parameter that can significantly affect the extraction results.

focal_length_mm

This parameter specifies the focal length in millimeters, which is crucial for accurate camera motion extraction. It is an advanced setting that should be adjusted based on the camera used to capture the input video.

sensor_width_mm

This parameter defines the sensor width in millimeters, which, along with the focal length, helps in accurately modeling the camera's optical characteristics. It is an advanced parameter that should be set according to the camera specifications.

max_dimension

This parameter limits the maximum dimension of the video frames used in the extraction process. It helps manage computational load and can be adjusted to balance performance and accuracy.

frame_step

This parameter determines the step size between frames used in the extraction process. A smaller step size can lead to more accurate results but requires more computational resources.

normalize_origin

This boolean parameter, when set to true, normalizes the origin of the extracted camera trajectory, making it easier to integrate into different environments.

motion_scale

This parameter scales the extracted motion, allowing users to adjust the intensity of the camera movements to suit their needs.

position_smoothing

This parameter applies smoothing to the position data of the extracted camera trajectory, helping to reduce noise and create a more stable motion path.

rotation_smoothing

Similar to position smoothing, this parameter smooths the rotation data, ensuring that the extracted camera movements are fluid and realistic.

simplify_keys

This boolean parameter, when enabled, simplifies the keyframes in the extracted camera trajectory, making it easier to work with in animation and editing software.

position_tolerance

This parameter sets the tolerance for position changes in the extracted trajectory, allowing users to control the sensitivity of the extraction process.

rotation_tolerance_deg

This parameter defines the tolerance for rotation changes in degrees, providing control over the sensitivity of the rotation extraction.

OmniCam Extractor Output Parameters:

motion_scene

This output parameter provides a canonical OmniCam motion scene, which includes the extracted camera trajectory and other cinematic motion metadata. It is a comprehensive representation of the camera's movement throughout the video, ready for use in digital environments.

solver_coverage

This parameter indicates the confidence level of the solver in extracting the camera trajectory. It ranges from 0.0 to 1.0, with higher values representing greater confidence in the accuracy of the extracted motion.

report

This output provides a detailed report of the extraction process, including information about the solver used and any issues encountered. It is useful for troubleshooting and understanding the extraction results.

OmniCam Extractor Usage Tips:

• Ensure your input video is a continuous shot without hard cuts to achieve the best extraction results.
• Experiment with different method options to find the solver that provides the most accurate results for your specific video.
• Use position_smoothing and rotation_smoothing to create more stable and realistic camera movements, especially if the input video is shaky.

OmniCam Extractor Common Errors and Solutions:

SolveError: OmniCam Extractor needs at least 2 usable frames.

• Explanation: This error occurs when the input video does not contain enough frames for the extraction process.
• Solution: Ensure that your input video has at least two frames and that they are usable for motion extraction.

BackendUnavailableError: <specific backend unavailable message>

• Explanation: This error indicates that the selected backend solver is not available or not installed.
• Solution: Check the availability of the backend solvers and install any missing ones. You can also switch to a different solver using the method parameter.