comfyui-muse Introduction
comfyui-muse adds a local AI chat panel to a ComfyUI node. You can brainstorm image and video prompts, rewrite tags, discuss reference material, and copy the results into your workflow without switching to a separate chat window.
Muse is a prompt-writing companion, not a prompt-processing node. It has no graph inputs or outputs: you copy text from the chat and paste it into the nodes where you need it. ComfyUI’s Queue Prompt ignores the Muse node, so the panel can stay on your canvas while you render.
You can chat through LM Studio, Ollama, or Muse’s Direct loader for GGUF models. Muse also keeps separate conversations, accepts reference files and media attachments, and helps ComfyUI and your chat model take turns using GPU memory.
How comfyui-muse Works
Think of Muse as a writing desk beside your ComfyUI canvas. You ask a local language model for help, refine its answer in conversation, then copy the wording you want into your generation workflow.
A typical session looks like this:
- Add Muse Chat from Add Node → utils → muse → Muse Chat, or search for “Muse Chat.”
- Choose LM Studio, Ollama, or Direct (GGUF). For LM Studio or Ollama, confirm the displayed address and use ⟳ to list available models. For Direct, select a GGUF model from a folder you have added in its settings.
- Type a request and press Enter. Use Shift+Enter for a new line. Muse loads the selected model when needed and displays its reply as it arrives.
- Copy a useful prompt into your ComfyUI graph. Keep chatting, start another conversation, or branch from an earlier message to explore a different idea.
For example, you could ask for five variations of a cinematic lighting prompt, attach a reference image to ask a vision-capable model about its style, or provide a shot clip to a video-capable model for an editing-prompt draft.
Muse can also help your GPU switch jobs. Before a chat message, it can free memory held by ComfyUI’s models; before a render, it can unload the chat model. This is especially useful when both tasks would otherwise compete for the same video memory, or VRAM.
comfyui-muse Features
Conversations and prompt writing
- Separate chats: Create, switch, rename, and delete conversations. Each chat retains its own history, system prompt, model, and settings, so you can keep projects or styles apart. Chats are saved and restored across restarts.
- Streaming answers: Read a reply as it is generated, or press Stop if you already have what you need.
- Message controls: Hover over messages to copy, regenerate, or delete them. You can also edit your own messages and resend them; doing so removes the later messages in that conversation and generates a new continuation.
- Chat branching: Fork a conversation from any message to test another direction without losing the original thread. For example, branch after a reference-image discussion to compare a moody treatment with a bright editorial one.
- Reply length: Set Max reply tokens separately for each chat. The default is 2048; increasing it can help when you request several complete prompts at once. Muse indicates when an answer reaches the limit.
- Usage meter: See current token use against the model’s context length, including the cost of active guide material. Tokens are the pieces of text the model uses to track a conversation; a longer chat and more reference material leave less room in its context.
The panel is draggable and resizable, with a frosted-glass appearance and a choice of eight accent colors. Your color choice is remembered.
System prompts and Guide Materials
A system prompt gives a chat standing instructions, such as “Write concise, camera-ready prompts.” Guide Materials add reusable reference content—perhaps a project style guide or naming convention—to every message in that chat.
Place .txt, .md, or .json files in ComfyUI/input/, then open Guide materials and select the files you want. Your selection is saved per chat. Muse reads the files again for each message, so changes to a guide on disk take effect without rebuilding the conversation. Missing files are skipped and flagged.
Image, video, and audio attachments
Attachments let you ask a suitable model about material beyond text. What the model can understand depends on its capabilities.
-
Images: Use + to select an image from
ComfyUI/input/, or drag an image onto the panel. Thumbnails appear above the message box, and sent images appear in the chat. Muse accepts formats Pillow can open, including PNG, JPEG, WebP, GIF, BMP, and TIFF; it converts less commonly supported formats to PNG before sending them. A vision-capable model is required. Try asking, “Describe the composition and lighting, then draft a prompt with a similar mood.” -
Videos: Use 🎬 to choose a video from
ComfyUI/input/, or drag one onto the panel. Muse extracts still frames, labels them with timestamps, and sends them with your question. Video sampling defaults to 1 frame per second, with max frames defaulting to 24. If a clip is longer than those settings would cover, Muse spreads the selected frames across the full duration rather than only showing its beginning. A video-capable vision model and OpenCV in ComfyUI’s Python environment are required. -
Audio: Use 🎤 or drag and drop an audio file to attach it to a message. For supported models through the Direct loader, Muse sends the original audio data without converting its format. At the time described by the author, Direct is the working end-to-end option for audio input; LM Studio and Ollama do not provide dependable audio-input support through the APIs Muse uses. Muse warns you about backend limitations and flags models whose names suggest audio capability.
Muse stores references to attached files in chat history rather than embedding the media itself in the saved conversation. It reads the files from ComfyUI/input/ when sending them, which keeps chat files smaller. Keep a referenced file in that folder if you want to use it again.
GPU memory coordination
Two top-bar switches are on by default:
- Free ComfyUI VRAM: Before sending each chat message, Muse unloads ComfyUI’s models and clears its cache to make room for the chat model.
- Unload on Run: Before a ComfyUI render starts, Muse unloads the chat model and checks that its memory has been freed. It does not reload the model after rendering; sending another message loads it when needed.
You can also press Unload whenever you want to free memory yourself. If you prefer to manage memory manually, you can change the two switches.
Direct model loader
Direct (GGUF) runs a compatible local chat model without keeping LM Studio or Ollama open. Muse starts llama.cpp’s llama-server to run the model, then talks to it through the same kind of chat interface it uses for LM Studio. This is why Direct is a separate way to run models, not a different kind of prompt-writing model.
In Direct Loader settings, you can:
- Add folders to scan recursively for
.gguffiles, or use Suggest folders to find common locations. - Use Download llama-server to fetch a matching build from the official llama.cpp releases. The default is a Vulkan build; advanced options include CPU-only and NVIDIA CUDA builds. Use existing install looks for a server already on your machine.
- Adjust GPU layers, which controls how much of the model runs on the GPU. The default,
-1, requests all layers. Fit to GPU suggests a value using available NVIDIA VRAM information. If loading still runs out of memory, Muse retries with progressively fewer GPU layers, down to CPU-only. - Adjust Context length, which controls how much conversation the model can keep available. The default is 8192 tokens. A much larger value can use substantially more memory, even before your chat grows long.
- Choose Flash attention (
autoby default, oron/off) and provide Extra args if you already know you need additionalllama-serveroptions.
Only one Direct model runs at a time; selecting another stops the previous process first. While a model loads, Muse shows elapsed time rather than an estimated percentage. Open Log to see persistent chat messages and, for Direct, the server’s live process output—particularly useful when a model cannot load.
comfyui-muse Models
Muse does not include its own chat models. You choose a model available through LM Studio or Ollama, or point Direct at GGUF model files you have on disk. Choose according to the task:
| Model type | When to use it | Example |
|---|---|---|
| Text chat model | Brainstorm, expand, shorten, or rewrite prompts and tag lists. | Ask for three variations of a product-photography prompt. |
| Vision-capable model | Discuss an attached image or sampled video frames. For video, choose a model capable of understanding visual sequences, such as Qwen3-VL. | Ask how the lighting changes across a clip. |
| Audio-capable model | Discuss or transcribe attached audio through the Direct loader. Examples named by the author include Qwen2-Audio, Qwen2.5-Omni, Ultravox, and Voxtral. | Ask about speech or sounds in a reference clip. |
A model name alone is not a guarantee that every attachment will work. For Direct vision and audio models, keep the model’s .gguf file and its matching mmproj-*.gguf projector file in the same folder; Muse pairs them automatically and marks the model’s media capability in the menu.
Direct also recognizes split GGUF models by their first file and filters out diffusion-model GGUF checkpoints it encounters in ComfyUI model folders. A Flux or SDXL checkpoint may share the GGUF file format, but it is not a chat model.
What's New with comfyui-muse
V2 adds the Direct GGUF loader, so you can chat without LM Studio or Ollama, as well as audio attachments, editing and resending messages, and branching conversations. These changes make it easier to reuse an existing discussion, correct a request, or test a new creative direction without starting over.
For the author’s fuller account of the V2 changes, see RELEASE_NOTES.md. The provided documentation does not describe a version-by-version history beyond V2.
Troubleshooting comfyui-muse
If something is not working, start with the model, backend, and attachment type you selected. The Log panel is especially useful for Direct-loader problems.
| Problem | What to try |
|---|---|
| No models appear | Confirm the selected backend. For LM Studio or Ollama, check that the app is running and its address in Muse is correct, then press ⟳. For Direct, check that you added a folder containing chat-model .gguf files. |
| A Direct model will not load | Open Log and read the load message. If memory is the issue, try Fit to GPU, fewer GPU layers, or a smaller context length. Muse may also retry automatically with less GPU offloading; the log shows the outcome. |
| Chat is unexpectedly slow | Check the Direct loader’s context length and GPU settings. An oversized context can exhaust VRAM or make generation much slower. The Log can help show whether the model fell back to fewer GPU layers or CPU operation. |
| An image or video is not understood | Confirm that you selected a vision-capable model. With Direct, check that its matching mmproj file is beside the model. For video, also check whether ComfyUI reports that OpenCV is missing. |
| Audio is ignored or a warning appears | Use the Direct backend with an audio-capable model and its matching projector file. LM Studio and Ollama audio-input support is limited in the APIs described by the author. |
| A reference file stops affecting replies | Check that it is still in ComfyUI/input/ and selected under Guide materials for this chat. Muse flags missing files. |
| A reply stops before it finishes | Increase that chat’s Max reply tokens, then ask again. The panel indicates when a reply hits the set limit. |
| Rendering runs out of GPU memory | Check that Unload on Run is enabled, or press Unload before rendering. To make room before chatting, enable Free ComfyUI VRAM. |
Frequently asked questions
Does Muse send its chat text directly into my ComfyUI workflow?
No. Muse intentionally has no graph connections. Copy the text you like and paste it into your workflow.
Will my conversations still be there after a restart?
Yes. Muse saves chats and restores them across restarts.
Do I need LM Studio or Ollama to use Muse?
No. The Direct loader can run a compatible GGUF chat model from disk using llama-server.
Why does the model load again after I render?
With Unload on Run enabled, Muse unloads the chat model so ComfyUI can use the VRAM. It loads again when you send a new message.
Can I ask about a whole video?
You can attach a video, but Muse sends selected, timestamped frames—not every frame. Increasing the sampling rate or frame limit can provide more visual detail, subject to the model’s available context and your machine’s resources.
Learn More about comfyui-muse
- V2 release notes — the author’s account of the updated features.
- llama.cpp — background on the local model engine used by Direct.
- Official llama.cpp releases — builds to consult if Muse cannot find a matching automatic download for your platform.
Muse is released under the MIT license.
