ComfyUI Node: SAM3 Text Segmentation

SAM3 Text Segmentation:

SAM3Grounding is a powerful node designed for text-based object detection within images, leveraging the capabilities of the SAM3 model. This node allows you to identify and segment objects in an image by providing a descriptive text prompt, such as "find all dogs." It enhances the detection process by optionally incorporating geometric prompts, like bounding boxes, to refine the search area or exclude certain regions. The node is particularly beneficial for AI artists and designers who wish to automate the process of identifying specific objects in their artwork or images without needing extensive technical knowledge. By setting a confidence threshold, you can control the precision of the detections, ensuring that only objects meeting a certain confidence level are considered. The SAM3Grounding node is an essential tool for those looking to integrate advanced object detection into their creative workflows, offering a seamless and intuitive way to interact with complex image data.

SAM3 Text Segmentation Input Parameters:

sam3_model

The sam3_model parameter refers to the SAM3ModelPatcher instance, which is essential for loading and managing the SAM3 model within the ComfyUI environment. This parameter ensures that the model is correctly loaded onto the GPU for efficient processing. There are no specific minimum, maximum, or default values for this parameter, as it is a required input for the node to function.

image

The image parameter is a ComfyUI image tensor with dimensions [B, H, W, C], representing the batch size, height, width, and color channels of the image. This parameter is crucial as it provides the visual data on which the text-based detection will be performed. The image should be in a format compatible with ComfyUI, and there are no specific constraints on its size or content.

confidence_threshold

The confidence_threshold parameter sets the minimum confidence score required for detections to be considered valid. This allows you to filter out less certain detections, ensuring that only objects with a confidence score above this threshold are included in the results. The value should be a float between 0 and 1, with a higher value indicating a stricter confidence requirement. There is no default value specified, so it should be set according to the desired level of detection accuracy.

text_prompt

The text_prompt parameter is a string that describes the objects you wish to detect within the image. This text-based input guides the SAM3 model in identifying relevant objects, making it a key component of the node's functionality. The text should be clear and descriptive, and there are no specific constraints on its length or content.

positive_boxes

The positive_boxes parameter is an optional input that allows you to specify bounding boxes to focus the detection process on certain areas of the image. This can enhance the accuracy of the detection by narrowing down the search area. The parameter should be a dictionary containing 'boxes' and 'labels', and it is optional, meaning it can be left empty if not needed.

negative_boxes

The negative_boxes parameter is similar to positive_boxes but serves the opposite purpose. It allows you to specify areas of the image to exclude from detection, helping to avoid false positives in regions where objects should not be detected. Like positive_boxes, it should be a dictionary with 'boxes' and 'labels', and it is optional.

max_detections

The max_detections parameter sets the maximum number of detections to return from the node. This allows you to limit the number of results, which can be useful for managing output size and focusing on the most relevant detections. The parameter should be an integer, and there is no default value specified, so it should be set according to your needs.

SAM3 Text Segmentation Output Parameters:

masks

The masks output parameter provides the segmentation masks for the detected objects. These masks are binary images that highlight the areas of the image corresponding to each detected object, allowing for precise segmentation and visualization of the results.

visualization

The visualization output parameter is a visual representation of the detection results, typically showing the original image with overlays of the detected objects and their corresponding masks. This output is useful for quickly assessing the accuracy and relevance of the detections.

boxes_json

The boxes_json output parameter contains the bounding boxes of the detected objects in JSON format. This structured data provides the coordinates and dimensions of each detected object, which can be used for further analysis or integration into other workflows.

scores_json

The scores_json output parameter provides the confidence scores for each detected object in JSON format. These scores indicate the model's confidence in each detection, allowing you to assess the reliability of the results and make informed decisions based on the confidence levels.

SAM3 Text Segmentation Usage Tips:

• Use clear and specific text prompts to improve the accuracy of object detection. The more descriptive the prompt, the better the model can identify the desired objects.
• Adjust the confidence_threshold to balance between precision and recall. A higher threshold will result in fewer, but more accurate detections, while a lower threshold may increase the number of detections but include more false positives.
• Utilize positive_boxes and negative_boxes to refine the detection process, especially in complex images with multiple objects. This can help focus the model's attention on relevant areas and exclude irrelevant ones.

SAM3 Text Segmentation Common Errors and Solutions:

"Model not loaded"

• Explanation: This error occurs when the SAM3 model is not properly loaded onto the GPU, which is necessary for processing.
• Solution: Ensure that the sam3_model parameter is correctly set and that the model is loaded using ComfyUI's model management system.

"Invalid image format"

• Explanation: The input image is not in the expected ComfyUI tensor format, which can prevent the node from processing it correctly.
• Solution: Convert your image to the ComfyUI tensor format [B, H, W, C] before passing it to the node.

"Text prompt is empty"

• Explanation: The text prompt provided is empty or consists only of whitespace, which means the model has no guidance for object detection.
• Solution: Provide a clear and descriptive text prompt to guide the detection process.

"Exceeded max detections"

• Explanation: The number of detected objects exceeds the specified max_detections limit.
• Solution: Increase the max_detections parameter if you need more results, or refine your text prompt and box parameters to focus on fewer objects.