ComfyUI Node: DP Broken Token

DP Broken Token:

The DP Broken Token node is a specialized tool within the ComfyUI framework designed to analyze and split text prompts based on user-defined token limits. This node is particularly useful for AI artists who need to manage and manipulate large text inputs by breaking them down into manageable parts. By leveraging the capabilities of the FluxTokenizer, the node efficiently processes text prompts, ensuring that each segment adheres to specified token constraints. This functionality is crucial for applications where precise token management is necessary, such as in natural language processing tasks or when interfacing with models that have strict input size limitations. The node not only splits the text but also provides detailed information about the token distribution across the segments, enhancing the user's ability to optimize and refine their text inputs for better performance and results.

DP Broken Token Input Parameters:

max_tokens_part_1

This parameter defines the maximum number of tokens allowed in the first part of the text prompt. It plays a critical role in determining how the initial segment of the text is split, impacting the distribution of content across the different parts. The minimum value is 0, the maximum is 2000, and the default is set to 77. Adjusting this parameter allows you to control the size of the first text segment, which can be useful for ensuring that the most important content is prioritized or fits within specific constraints.

max_tokens_part_2

Similar to max_tokens_part_1, this parameter sets the maximum token limit for the second part of the text prompt. It influences how the middle section of the text is divided, affecting the overall structure and flow of the prompt. The minimum value is 0, the maximum is 2000, and the default is 0. By configuring this parameter, you can manage the allocation of tokens to the second segment, which is beneficial for maintaining coherence and relevance in the text output.

prompt

This optional parameter allows you to input the text prompt that you wish to analyze and split. It supports multiline text and is designed to be user-friendly, with options for default input and forced input to ensure that the prompt is correctly processed. The prompt serves as the primary input for the node's analysis, and its content will be divided into parts based on the specified token limits.

DP Broken Token Output Parameters:

total_tokens

This output provides the total number of tokens present in the entire text prompt after analysis. It is a crucial metric for understanding the overall size of the input and ensuring that it aligns with any constraints or requirements you may have.

info

The info output delivers a detailed string containing information about the token distribution across the different parts of the text. It includes the total token count and the number of tokens in each segment, offering valuable insights into how the text has been split and allowing for further adjustments if necessary.

Text part 1

This output contains the first segment of the text prompt, split according to the max_tokens_part_1 parameter. It represents the initial portion of the text and is essential for tasks that require specific content to be prioritized or highlighted.

Text part 2

The second segment of the text prompt is provided in this output, divided based on the max_tokens_part_2 parameter. It forms the middle section of the text and is important for maintaining the continuity and flow of the content.

Text part 3

This output includes the remaining portion of the text prompt that exceeds the limits set by the first two parameters. It captures any additional content that could not be accommodated in the initial segments, ensuring that no part of the input is lost.

DP Broken Token Usage Tips:

• To optimize the node's performance, carefully set the max_tokens_part_1 and max_tokens_part_2 parameters based on the importance and priority of different sections of your text prompt.
• Use the info output to gain insights into the token distribution and make necessary adjustments to your input parameters for better alignment with your specific needs.
• When dealing with large text inputs, consider breaking them down into smaller, more manageable parts using this node to ensure compatibility with models that have strict input size limitations.

DP Broken Token Common Errors and Solutions:

Invalid token count

• Explanation: This error may occur if the specified token limits exceed the maximum allowed values or are set to negative numbers.
• Solution: Ensure that the max_tokens_part_1 and max_tokens_part_2 parameters are within the valid range of 0 to 2000.

Prompt not provided

• Explanation: If no prompt is provided, the node returns default values, which may not be the desired outcome.
• Solution: Always provide a valid text prompt to ensure that the node can perform the analysis and splitting as intended.