ComfyUI-TaoMateHD Introduction
ComfyUI-TaoMateHD is an unofficial ComfyUI extension that brings TaoMate-HD’s three-stage, progressive-resolution sampler to native MiniMax H3 audio-video models.
It helps you create short videos with accompanying audio, either from text or, experimentally, with a reference image. It also includes digital-human workflows that preserve an existing voice recording while generating the picture.
Instead of performing every generation step at full resolution, the sampler starts with a smaller video canvas and moves to the final resolution for the last stage. Think of it as establishing a rough composition before adding detail.
The extension provides:
- Text-to-video-and-audio generation (T2VA).
- Experimental reference-to-video-and-audio generation (Ref2VA).
- A dedicated sampler node that handles TaoMate-HD’s three stages internally.
- Audio and video masks for controlling which parts are generated or preserved.
- Digital-human examples for working with a reference image and source audio.
- An optional SOL block-sparse attention workflow for accelerated experimentation. Important scope: this adapter is currently a single-window proof of concept, designed around approximately five seconds of output. The upstream TaoMate-HD project supports long-form streaming, but that streaming system is not included in this ComfyUI extension.
Extension repository · Converted ComfyUI LoRA · Comfy Registry entry
Digital-human demonstration
[TaoMate-HD with SOL digital-human demonstration](demo/taomate_hd_digital_human_sol.mp4)
The supplied SOL showcase is 1920 × 1088 at 24 FPS, lasting 5.17 seconds. Click the preview to open the MP4 with audio. Its dimensions describe this demonstration, not a guaranteed performance target for every GPU.
How ComfyUI-TaoMateHD Works
From a rough canvas to a detailed video
The TaoMate HD Progressive Sampler performs three model evaluations inside one ComfyUI node:
| Stage | Video resolution | Purpose |
|---|---|---|
| First stage | Lower resolution | Establish the initial audio-video structure. |
| Second stage | Lower resolution | Continue refining that structure. |
| Resolution transition | Lower to higher resolution | Move the video representation onto the final-resolution grid using coupled noise. |
| Third stage | Final resolution | Complete generation at the target size. |
The resolution transition is not an additional model evaluation. With the default steps=3, you get two lower-resolution evaluations and one final-resolution evaluation.
The fixed teacher-state boundaries are 0, 16, 33, and 49. These are internal schedule positions—not a request for 49 sampling steps.
What is a latent?
A latent is a compact working representation of the video or audio. The model generates and refines this representation before a decoder turns it into viewable frames or playable sound.
This extension expects MiniMax H3’s native combined audio-video latent, rather than an ordinary image-only latent.
After sampling:
- Separate AV Latent separates the video and audio representations.
- VAEDecode, using the video VAE, turns the video latent into frames.
- VAEDecodeAudio, using the audio VAE, turns the audio latent into sound. A VAE is the model component that translates between media and its compact latent representation.
Why coupled noise matters
Generation begins from noise. TaoMate-HD coordinates the noise used for the lower- and higher-resolution video stages, rather than switching to an unrelated noise pattern when the canvas grows.
An artist-friendly analogy is transferring the same sketch onto a larger canvas instead of starting a new sketch halfway through. The adapter also couples video and audio noise, while keeping the audio latent’s geometry unchanged across stages.
How reference images and source audio fit in
For reference-image workflows, the adapter carries H3’s native minimax_refs reference information through all three stages.
For source-audio preservation, a noise mask marks the audio as something to retain:
0means preserve.1means generate. The supplied digital-human workflows preserve the source voice and leave the video available for generation.
ComfyUI-TaoMateHD Features
Text-to-video-and-audio generation
T2VA creates video and audio from text conditioning.
The supplied T2VA workflow uses MiniMax H3 Image to Video without first or last frames. Although the conditioning node’s name includes “Image to Video,” no image endpoints are supplied in this example.
Connect its outputs as follows:
MiniMax H3 Image to Video — no first/last frames
positive -> TaoMateHDProgressiveSampler.positive
latent -> TaoMateHDProgressiveSampler.latent
Experimental reference-image generation
Ref2VA adds a reference image through MiniMax H3 Reference to Video.
MiniMax H3 Reference to Video
positive -> TaoMateHDProgressiveSampler.positive
latent -> TaoMateHDProgressiveSampler.latent
Use a Ref2VA-capable or hybrid H3 checkpoint for this workflow.
This feature is experimental: the author uses H3’s native reference-conditioning path, rather than the T2VA-only public TaoMate-HD inference script. Treat reference-image results as an experimental capability, not a guaranteed match to upstream behavior.
Source-audio preservation and digital humans
The digital-human examples combine a reference image with an existing audio recording.
Their key audio-preservation path is:
Encoded source-audio latent
-> SetLatentNoiseMask with SolidMask = 0
-> Combined audio-video latent
-> TaoMateHDProgressiveSampler
With the audio mask set to 0, the sampler preserves the source voice to support the lip-sync workflow. The video latent remains unmasked so H3 can generate the picture.
Changing the audio mask to 1 allows audio generation instead of preserving the supplied recording.
Optional SOL workflow
The SOL example adds:
BlockSparseAttentionfromsol-attn.QwenH3PromptLocalfor local prompt enhancement.ModelPreviewOverrideKJandPreviewAnyfor previews. This is an optional accelerated digital-human example. Its supporting custom-node packs must be available in your ComfyUI environment.
SOL acceleration is not the upstream eight-GPU pipeline. They are separate approaches, and the extension does not promise the upstream project’s performance.
Sampler controls
| Input | What it controls | Recommended starting point |
|---|---|---|
model | The MiniMax H3 model with the converted TaoMate-HD LoRA applied. | Use a compatible H3 model and the converted adapter. |
positive | Text and, where applicable, reference-image conditioning. | Use the output of the appropriate H3 conditioning node. |
negative | Negative conditioning used with higher CFG. | Required only when cfg > 1.0. |
latent | The native H3 audio-video working representation. | Use the H3 conditioning node’s latent output or a compatible empty H3 AV latent. |
seed | The coupled video/audio noise seed. | Keep it fixed when comparing settings; change it to explore variations. |
steps | Total model evaluations across the stages. | 3, the official distilled schedule. |
cfg | Classifier-free guidance strength. | 1.0, which TaoMate is distilled for. |
low_scale | The spatial scale of the lower-resolution stages. | 0.625, equivalent to 480/768. |
debug | Diagnostic logging. | Enable when investigating shapes, stage allocation, or resolution transitions. |
A fixed seed helps make comparisons more controlled, but it does not imply identical output after changing models or other settings.
Experimental step allocations
Higher step counts subdivide the existing stages; they do not add new progressive-resolution stages.
steps | First low-resolution stage | Second low-resolution stage | High-resolution stage |
|---|---|---|---|
3 | 1 | 1 | 1 |
4 | 2 | 1 | 1 |
5 | 2 | 2 | 1 |
6 | 2 | 2 | 2 |
8 | 3 | 3 | 2 |
Start with 3. Higher values require more model evaluations and are experimental, not an assurance of better quality.
ComfyUI-TaoMateHD Models
The extension does not provide a collection of separate style models. It combines a compatible MiniMax H3 base model with a TaoMate-HD LoRA adapter and the necessary encoding/decoding components.
Model components
| Component | Role | When to use it |
|---|---|---|
| MiniMax H3 model with FL2VA capabilities | Provides the underlying H3 generation model. | For compatible text-conditioned workflows. FL2VA first/last-frame controls are not implemented by this adapter. |
| Ref2VA-capable or hybrid MiniMax H3 checkpoint | Supports native reference-image conditioning. | For the reference-image and digital-human examples. |
TaoMate-HD-preview-comfy.safetensors | The converted LoRA that adapts the base model for TaoMate-HD sampling. | Apply it to the H3 model before the progressive sampler. |
| MiniMax H3 text encoder | Converts prompt text into model conditioning. | Required for text conditioning. |
| MiniMax H3 video VAE | Decodes the video latent into frames. | Required for video output. |
| MiniMax H3 audio VAE | Encodes/decodes audio as required by the workflow. | Required for audio handling. |
Resources:
- MiniMax H3 models
- ComfyUI-compatible TaoMate-HD preview LoRA
- Original upstream TaoMate-HD preview adapter
Choosing the correct LoRA
Use the converted file with:
LoraLoaderModelOnly
lora_name = TaoMate-HD-preview-comfy.safetensors
strength_model = 1.0
The converted file uses ComfyUI-compatible keys and includes TaoMate’s per-module alpha values, which help preserve the adapter’s intended scaling.
Do not load the original upstream adapter_model.safetensors directly in ComfyUI.
The default conversion produces FP32 weights of approximately 2.48 GB. A smaller BF16 conversion is available as an experimental option; it is not documented here as a separate quality model.
Hardware expectations
The upstream implementation reports validation on eight H20 GPUs with 96 GB each. This adapter instead reuses native H3 loaders and quantized models to make single-GPU, single-window experiments possible.
That does not guarantee a particular memory requirement or speed.
For a 32 GB GPU, the supplied guidance is to:
- Start with an int8 or fp8 H3 model.
- Keep the text encoder offloadable so it can leave GPU memory when not needed.
- Test at 480p before trying 768p.
ComfyUI-TaoMateHD Workflows
Core node sequence
UNETLoader
-> LoraLoaderModelOnly
-> MiniMaxH3SigmaShift
shift_video = 12
shift_audio = 3
-> TaoMate HD Progressive Sampler
-> Separate AV Latent
-> VAEDecode with video VAE
-> VAEDecodeAudio with audio VAE
Supplied examples
| Workflow | Purpose |
|---|---|
taomateHD_t2va.json | Text-conditioned audio-video generation. |
taomateHD_ref2va.json | Experimental reference-image generation. |
TaoMateHD_digital_human.json | Standard digital-human workflow with source-audio preservation. |
TaoMateHD_digital_human_SOL.json | SOL digital-human workflow with prompt enhancement and previews. |
Before running an example:
- Select your own compatible model files in the model-loading nodes.
- For reference-image examples, choose an image in
LoadImage. - For digital-human examples, also choose a recording in
LoadAudio. - Check the resolution, duration, and frame-length settings before generating. The workflows contain ComfyUI model names, not bundled model files. Their selections must match files available in your environment.
Resolution and timing
Both generation dimensions must be multiples of 32.
The standard digital-human example uses:
- A 16:9, approximately 1 MP resolution selection.
ResolutionSelector.multiple = 32.- A 1376 × 768 canvas.
- An 86 × 48 H3 latent grid. The alignment keeps the latent axes even for H3’s 2 × 2 patch processing.
A PrimitiveFloat controls the example’s five-second source-audio duration and also feeds its 124-frame length calculation. Do not change duration independently without checking that audio and video lengths still agree.
What's New with ComfyUI-TaoMateHD
The supplied documentation records one dated adapter update and several additional changes without individual release dates.
| Update | Change | Why it matters |
|---|---|---|
| 2026-11-07 — Canonical digital-human workflow | Earlier digital-human drafts were replaced with examples/TaoMateHD_digital_human.json. | You have a clearly identified standard example to start from. |
| AV latent masks | The sampler now accepts MiniMax H3 video/audio noise masks. | You can preserve source audio with 0 or allow generation with 1. |
| Resolution alignment guidance | H3 dimensions must be 32-pixel aligned; the digital-human example uses multiple=32. | Helps prevent latent-grid and patch-processing shape problems. |
| SOL digital-human workflow | Added block-sparse attention, local prompt enhancement, and preview nodes. | Provides an optional accelerated workflow to explore. |
| SOL demonstration | Added a GIF preview and an MP4 with audio. | Lets you inspect a supplied example of the workflow’s output. |
For upstream context, TaoMate-HD’s inference code, preview LoRA, and paper were released on 2026-11-03. These upstream releases should not be confused with a complete version history for the ComfyUI adapter.
ComfyUI-TaoMateHD Limitations
The current adapter supports:
-
A single approximately five-second window.
-
Approximately 124 frames at 24 FPS.
-
Batch size 1.
-
T2VA generation.
-
Experimental native H3 Ref2VA conditioning.
-
Audio/video latent masks. It does not include:
-
FL2VA first/last-frame keyframe controls.
-
A loop for consecutive five-second windows.
-
Cross-window, stage-local history caching.
-
The upstream eight-GPU
1/1/4/2generation-and-decoding pipeline. The upstream project demonstrates long-form streams, but those results should not be interpreted as features already available in this adapter.
Troubleshooting ComfyUI-TaoMateHD
The workflow reports an incompatible LoRA or model
- Check that the base model is a compatible MiniMax H3 model.
- Confirm that the LoRA is
TaoMate-HD-preview-comfy.safetensors, not the original upstream adapter file. - Load it through
LoraLoaderModelOnlyatstrength_model=1.0. - For reference workflows, select a Ref2VA-capable or hybrid checkpoint.
A resolution or latent-shape error appears
- Check that both width and height are divisible by 32.
- If using
ResolutionSelector, setmultiple=32. - Return to the supplied example’s resolution before testing custom dimensions.
- Enable
debugto inspect stage shapes and resolution-transition details.
My source voice is being regenerated
- Verify that the intended recording is selected in
LoadAudio. - Check that the encoded audio latent passes through
SetLatentNoiseMask. - Set its
SolidMaskvalue to0. - Confirm that this masked audio latent is included in the AV latent sent to the sampler.
- Leave the video latent unmasked if you want H3 to generate the picture.
Audio and video lengths no longer match
- Return to the supplied five-second example.
- Check the
PrimitiveFloatduration value. - Confirm that the same value feeds the frame-length calculation.
- Review both audio and frame lengths before testing a new duration. Longer durations are not a substitute for the upstream streaming implementation.
The SOL workflow contains missing nodes
- Check which nodes ComfyUI marks as unavailable.
- Compare them with the SOL dependencies:
BlockSparseAttention,QwenH3PromptLocal,ModelPreviewOverrideKJ, andPreviewAny. - Make sure their corresponding custom-node packs are available.
- Use the standard digital-human workflow if you do not need the SOL additions.
Generation runs out of GPU memory
- Test at 480p before attempting 768p.
- Use an int8/fp8 H3 model where supported by your setup.
- Keep the text encoder offloadable.
- Keep batch size at 1 and use the supported single-window workflow.
- Do not use the upstream eight-GPU results to predict single-GPU memory use or speed.
Negative conditioning is requested
Check cfg:
- At the intended
cfg=1.0, negative conditioning is optional. - With
cfg > 1.0, supply negative conditioning. Start withcfg=1.0, since TaoMate is distilled for that setting.
Frequently asked questions
Will more steps always look better? No. Three steps are the official distilled schedule. Higher settings only subdivide the stages and remain experimental.
Can I make an hour-long video with this extension? Not with the documented adapter implementation. Long-form streaming belongs to the upstream system.
Can I use a reference image? Yes, experimentally, with the supplied Ref2VA workflow and a compatible checkpoint.
Can I specify the first and last frames? Not through this adapter’s current implementation.
Is there a fixed list of allowed resolutions? The underlying approach does not require a fixed whitelist, but H3 dimensions must follow the 32-pixel grid. Hardware limits still apply.
Does the SOL workflow guarantee real-time generation? No. The documentation provides an accelerated example, not a universal speed guarantee.
Learn More about ComfyUI-TaoMateHD
Practical resources
- Example workflows: the best starting point for exploring text generation, references, and source-audio preservation.
- Extension repository: the author’s implementation, documentation, and examples.
- Repository issues: a place to review reported problems or describe a reproducible issue. Include your workflow, model selections, dimensions, and relevant error or debug output.
- Converted LoRA repository: the adapter weights intended for ComfyUI.
- Comfy Registry page: the extension’s registry listing.
Background and research
- TaoMate-HD project page: an overview of the upstream research and demonstrations.
- Upstream TaoMate-HD repository: details about progressive generation, streaming, and multi-GPU execution.
- TaoMate-HD paper: the research explanation and evaluation results.
- MiniMax H3 model repository: information about the underlying model family. Upstream benchmark numbers describe their stated hardware and execution setup. They are not measurements of this ComfyUI adapter.
ComfyUI-TaoMateHD License and Attribution
This is an unofficial adaptation and is not affiliated with Alibaba TaoLive AIGC or MiniMax.
The adapted implementation follows the MiniMax H3 Community License Agreement. Review LICENSE and NOTICE for the applicable terms and attribution.
The base model and upstream adapter remain subject to their original licenses. Review those terms before using or redistributing the model components.
