Y7 T5 Token Counter:
Y7 T5 Token Counter helps you check how a prompt is divided into T5 tokens and whether its token count exceeds a selected limit. It counts the full text, reports the count in the node’s on-screen result, and can display individual tokens, including those beyond the limit. This makes it useful when refining a long prompt: you can see how much of it fits and inspect the point where truncation would occur. The count uses a T5 tokenizer loaded from local files when available, with a Hugging Face download fallback. The node reports truncation for inspection; it does not shorten the text it passes onward.
Y7 T5 Token Counter Input Parameters:
text_in
The text you want to count. This is an optional STRING input with a default of an empty string; the source specifies no minimum or maximum text length. Connect the prompt you want to inspect. Before counting or passing it onward, the node replaces em dashes, en dashes, and the mathematical minus sign with a standard hyphen, so those characters will not be preserved exactly. If the input is empty, the node displays No input text provided. and passes an empty string onward.
Y7 T5 Token Counter Output Parameters:
text_out
A STRING containing the input text after the dash and minus-sign replacements described above. Connect this output to the next node to use the text while inspecting its token count here. Despite the node’s name, this output is not a number or a token list: the count and optional token listing appear in the node’s on-screen result, not in text_out. The output is not truncated when the reported count exceeds the limit.
Y7 T5 Token Counter Usage Tips:
- Connect the prompt to
text_inand read the on-screenToken Count: count / limitresult before sending the text to a downstream node. A positiveover limitfigure tells you how many tokens the full tokenized text exceeds the selected limit. - If the frontend provides a token-limit setting, set it to the limit you want to inspect. The node reads that setting from saved workflow widget information and falls back to 512 when the required information is missing; the token limit is not an input socket.
- Enable the frontend’s show-tokens setting when you need to inspect individual token pieces or the overflow section. Leave it off when the count alone is sufficient; the node skips formatting the token listing in that case. If available, adjust the tokens-per-line setting for a more readable listing. Neither setting is an input socket.
- Check the on-screen
Last Tokencontext when it appears to see text near the selected limit. Remember that the truncated token listing includes an end-of-sequence token, whiletext_outstill carries the full, untruncated text.
Y7 T5 Token Counter Common Errors and Solutions:
No input text provided.
- Explanation: The input is empty, so there is no text to tokenize.
- Solution: Connect or provide nonempty text to
text_inif you want a count.
Unable loading local tokenizer: files may be missing or corrupt.
- Explanation: The node could not load its local T5 tokenizer files and will try the configured Hugging Face fallback.
- Solution: If the fallback succeeds, you can continue using the node. If it also fails, check that the environment can access
google/t5-v1_1-xxlor restore the local tokenizer files.
Error saving tokenizer locally: ...
- Explanation: The fallback tokenizer was obtained, but saving it to the local tokenizer directory failed. The node then tries loading the fallback tokenizer again.
- Solution: Check that the local directory is writable. If the node still fails, check access to the fallback model; the text following the message may identify the save problem.
Error: extra_pnginfo is empty
- Explanation: Workflow metadata was unavailable when the node tried to write its on-screen result back to the workflow’s widget values. The node may still return
text_out. - Solution: Run the node within a saved or normally executed workflow and check whether its on-screen result updates.
Error: extra_pnginfo is not a dict or missing 'workflow' key
- Explanation: The supplied metadata does not have the workflow structure needed to update the node’s widget values.
- Solution: Reopen or save the workflow and run it again. If this occurs in a custom integration, ensure it supplies workflow metadata in the expected structure.
⚠️ extra_pnginfo is None or empty / ⚠️ 'workflow' key not found in extra_pnginfo
- Explanation: The node cannot read the frontend token-limit setting from workflow metadata, so it uses the fallback limit of 512. - Solution: Check that the workflow is providing metadata before relying on a custom token limit.
⚠️ No node found with ID: ... / ⚠️ Node is missing 'widgets_values' / ⚠️ Node is missing 'properties'
- Explanation: The node cannot find its saved workflow entry or the widget information needed to read the token limit. It uses 512 instead.
- Solution: Save and rerun the workflow. If you edited the workflow data outside the interface, check that this node and its widget information are present.
⚠️ Property 'token_limit_index' not found in node['properties'] / ⚠️ token_limit_index ... out of bounds for widgets_values (...)
- Explanation: The saved workflow does not point to a valid token-limit widget value, so the node uses 512. - Solution: Recreate or reconfigure the node’s frontend token-limit setting, then save and rerun the workflow.
⚠️ Value at index ... is not an int: ...
- Explanation: The saved token-limit value cannot be read as a whole number, so the node uses 512.
- Solution: Enter a valid whole-number token limit in the frontend setting, then rerun the workflow.
