ComfyUI Node: Build AutoTuner Python Evaluator

Build AutoTuner Python Evaluator:

The BuildAutoTunerPythonEvaluator node is designed to facilitate the integration of external Python-based evaluators into the LoRA AutoTuner framework. This node allows you to specify a Python callable that can be used to evaluate and rank candidates based on custom criteria, such as prompt or reference scoring. By leveraging this node, you can enhance the flexibility and precision of the AutoTuner by incorporating external logic and scoring mechanisms. This capability is particularly beneficial for AI artists who wish to tailor the evaluation process to specific artistic goals or datasets, thereby optimizing the performance and relevance of the LoRA models being tuned.

Build AutoTuner Python Evaluator Input Parameters:

module_path

The module_path parameter specifies the file path or importable module name that contains the evaluator callable. This parameter is crucial as it directs the node to the correct Python file or module where the evaluation logic is implemented. It must be a valid path or module name, and it is required for the node to function correctly. This parameter does not have a default value and must be provided by the user.

callable_name

The callable_name parameter indicates the specific function within the module that will be used for evaluation. This function should follow a specific signature, accepting parameters like model, clip, lora_data, config, context, and analysis_summary, and returning either a float score or a dictionary with a score and additional details. The default value for this parameter is "evaluate_candidate", and it is essential for identifying the correct function to execute within the specified module.

combine_mode

The combine_mode parameter determines how the AutoTuner combines the built-in score with the score from the external evaluator. It offers options such as "blend", "external_only", and "multiply", with "blend" being the default. This parameter allows you to control the influence of the external evaluator on the final scoring, providing flexibility in how different evaluation criteria are weighted.

weight

The weight parameter is a floating-point value that specifies the blend weight for the external evaluator when combine_mode is set to "blend". It ranges from 0.0 to 1.0, with a default value of 0.5. This parameter allows you to adjust the relative importance of the external evaluator's score in the overall evaluation process, enabling fine-tuning of the scoring balance.

context_json

The context_json parameter is a string representing an optional JSON object that is passed to the evaluator as context. This context can include additional information or parameters that the evaluator might need to perform its task. The default value is an empty JSON object ("{}"), and it supports multiline input for more complex JSON structures. This parameter enhances the evaluator's ability to consider contextual information during the evaluation process.

Build AutoTuner Python Evaluator Output Parameters:

evaluator

The evaluator output parameter is an object that encapsulates the specification for the Python evaluator. It includes details such as the module path, callable name, combine mode, weight, and context. This output is crucial as it defines the configuration and logic that the AutoTuner will use to evaluate and rank candidates, allowing for customized and potentially more effective tuning processes.

Build AutoTuner Python Evaluator Usage Tips:

• Ensure that the module_path and callable_name are correctly specified to avoid import errors. Double-check the path and function name for typos or incorrect references.
• Use the combine_mode and weight parameters to fine-tune the influence of the external evaluator. Experiment with different settings to achieve the desired balance between internal and external scoring.
• Leverage the context_json parameter to pass additional information to the evaluator, which can be useful for context-aware evaluations or when specific conditions need to be considered.

Build AutoTuner Python Evaluator Common Errors and Solutions:

"module_path and callable_name are required"

• Explanation: This error occurs when either the module_path or callable_name is not provided, which are essential for locating and executing the evaluator function.
• Solution: Ensure that both module_path and callable_name are specified correctly in the input parameters.

"Unable to load evaluator module from <module_path>"

• Explanation: This error indicates that the specified module could not be loaded, possibly due to an incorrect path or missing file.
• Solution: Verify that the module_path is correct and that the file exists at the specified location. Check for any typos or incorrect directory paths.

"Callable '<callable_name>' not found in <module_path>"

• Explanation: This error means that the specified function could not be found within the module, possibly due to a typo or incorrect function name.
• Solution: Double-check the callable_name to ensure it matches the function name in the module. Ensure that the function is defined and accessible within the specified module.