Save 4 hours! We auto-setup your workflow! Free!

Drop your workflow.json — we handle every dependency, custom node, and model. Just open the link and run.

Auto-Setup Workflow Json (Free) Now!
ComfyUI > Nodes > comfyui-muse

ComfyUI Extension: comfyui-muse

Repo Name

comfyui-muse

Author
RudySen (Account age: 991 days)
Nodes
View all nodes(1)
Latest Updated
2026-08-17
Github Stars
0.04K

How to Install comfyui-muse

Install this extension via the ComfyUI Manager by searching for comfyui-muse
  • 1. Click the Manager button in the main menu
  • 2. Select Custom Nodes Manager button
  • 3. Enter comfyui-muse in the search bar
After installation, click the Restart button to restart ComfyUI. Then, manually refresh your browser to clear the cache and access the updated list of nodes.

Visit ComfyUI Online for ready-to-use ComfyUI environment

  • Free trial available
  • 16GB VRAM to 80GB VRAM GPU machines
  • 400+ preloaded models/nodes
  • Freedom to upload custom models/nodes
  • 200+ ready-to-run workflows
  • 100% private workspace with up to 200GB storage
  • Dedicated Support

Run ComfyUI Online

comfyui-muse Description

comfyui-muse adds a local AI chat panel to ComfyUI for brainstorming and refining image or video prompts from text and media references. Copy results into your workflow while it manages GPU memory.

comfyui-muse Introduction

comfyui-muse adds a local AI chat panel to a ComfyUI node. You can brainstorm image and video prompts, rewrite tags, discuss reference material, and copy the results into your workflow without switching to a separate chat window.

Muse is a prompt-writing companion, not a prompt-processing node. It has no graph inputs or outputs: you copy text from the chat and paste it into the nodes where you need it. ComfyUI’s Queue Prompt ignores the Muse node, so the panel can stay on your canvas while you render.

Muse Chat generating prompts

You can chat through LM Studio, Ollama, or Muse’s Direct loader for GGUF models. Muse also keeps separate conversations, accepts reference files and media attachments, and helps ComfyUI and your chat model take turns using GPU memory.

How comfyui-muse Works

Think of Muse as a writing desk beside your ComfyUI canvas. You ask a local language model for help, refine its answer in conversation, then copy the wording you want into your generation workflow.

A typical session looks like this:

  1. Add Muse Chat from Add Node → utils → muse → Muse Chat, or search for “Muse Chat.”
  2. Choose LM Studio, Ollama, or Direct (GGUF). For LM Studio or Ollama, confirm the displayed address and use ⟳ to list available models. For Direct, select a GGUF model from a folder you have added in its settings.
  3. Type a request and press Enter. Use Shift+Enter for a new line. Muse loads the selected model when needed and displays its reply as it arrives.
  4. Copy a useful prompt into your ComfyUI graph. Keep chatting, start another conversation, or branch from an earlier message to explore a different idea.

For example, you could ask for five variations of a cinematic lighting prompt, attach a reference image to ask a vision-capable model about its style, or provide a shot clip to a video-capable model for an editing-prompt draft.

Muse can also help your GPU switch jobs. Before a chat message, it can free memory held by ComfyUI’s models; before a render, it can unload the chat model. This is especially useful when both tasks would otherwise compete for the same video memory, or VRAM.

Panel overview

comfyui-muse Features

Conversations and prompt writing

  • Separate chats: Create, switch, rename, and delete conversations. Each chat retains its own history, system prompt, model, and settings, so you can keep projects or styles apart. Chats are saved and restored across restarts.
  • Streaming answers: Read a reply as it is generated, or press Stop if you already have what you need.
  • Message controls: Hover over messages to copy, regenerate, or delete them. You can also edit your own messages and resend them; doing so removes the later messages in that conversation and generates a new continuation.
  • Chat branching: Fork a conversation from any message to test another direction without losing the original thread. For example, branch after a reference-image discussion to compare a moody treatment with a bright editorial one.
  • Reply length: Set Max reply tokens separately for each chat. The default is 2048; increasing it can help when you request several complete prompts at once. Muse indicates when an answer reaches the limit.
  • Usage meter: See current token use against the model’s context length, including the cost of active guide material. Tokens are the pieces of text the model uses to track a conversation; a longer chat and more reference material leave less room in its context.

The panel is draggable and resizable, with a frosted-glass appearance and a choice of eight accent colors. Your color choice is remembered.

System prompts and Guide Materials

A system prompt gives a chat standing instructions, such as “Write concise, camera-ready prompts.” Guide Materials add reusable reference content—perhaps a project style guide or naming convention—to every message in that chat.

Place .txt, .md, or .json files in ComfyUI/input/, then open Guide materials and select the files you want. Your selection is saved per chat. Muse reads the files again for each message, so changes to a guide on disk take effect without rebuilding the conversation. Missing files are skipped and flagged.

Image, video, and audio attachments

Attachments let you ask a suitable model about material beyond text. What the model can understand depends on its capabilities.

  • Images: Use + to select an image from ComfyUI/input/, or drag an image onto the panel. Thumbnails appear above the message box, and sent images appear in the chat. Muse accepts formats Pillow can open, including PNG, JPEG, WebP, GIF, BMP, and TIFF; it converts less commonly supported formats to PNG before sending them. A vision-capable model is required. Try asking, “Describe the composition and lighting, then draft a prompt with a similar mood.”

    Image attachment with a vision model

  • Videos: Use 🎬 to choose a video from ComfyUI/input/, or drag one onto the panel. Muse extracts still frames, labels them with timestamps, and sends them with your question. Video sampling defaults to 1 frame per second, with max frames defaulting to 24. If a clip is longer than those settings would cover, Muse spreads the selected frames across the full duration rather than only showing its beginning. A video-capable vision model and OpenCV in ComfyUI’s Python environment are required.

  • Audio: Use 🎤 or drag and drop an audio file to attach it to a message. For supported models through the Direct loader, Muse sends the original audio data without converting its format. At the time described by the author, Direct is the working end-to-end option for audio input; LM Studio and Ollama do not provide dependable audio-input support through the APIs Muse uses. Muse warns you about backend limitations and flags models whose names suggest audio capability.

Muse stores references to attached files in chat history rather than embedding the media itself in the saved conversation. It reads the files from ComfyUI/input/ when sending them, which keeps chat files smaller. Keep a referenced file in that folder if you want to use it again.

GPU memory coordination

Two top-bar switches are on by default:

  • Free ComfyUI VRAM: Before sending each chat message, Muse unloads ComfyUI’s models and clears its cache to make room for the chat model.
  • Unload on Run: Before a ComfyUI render starts, Muse unloads the chat model and checks that its memory has been freed. It does not reload the model after rendering; sending another message loads it when needed.

You can also press Unload whenever you want to free memory yourself. If you prefer to manage memory manually, you can change the two switches.

Direct model loader

Direct (GGUF) runs a compatible local chat model without keeping LM Studio or Ollama open. Muse starts llama.cpp’s llama-server to run the model, then talks to it through the same kind of chat interface it uses for LM Studio. This is why Direct is a separate way to run models, not a different kind of prompt-writing model.

In Direct Loader settings, you can:

  • Add folders to scan recursively for .gguf files, or use Suggest folders to find common locations.
  • Use Download llama-server to fetch a matching build from the official llama.cpp releases. The default is a Vulkan build; advanced options include CPU-only and NVIDIA CUDA builds. Use existing install looks for a server already on your machine.
  • Adjust GPU layers, which controls how much of the model runs on the GPU. The default, -1, requests all layers. Fit to GPU suggests a value using available NVIDIA VRAM information. If loading still runs out of memory, Muse retries with progressively fewer GPU layers, down to CPU-only.
  • Adjust Context length, which controls how much conversation the model can keep available. The default is 8192 tokens. A much larger value can use substantially more memory, even before your chat grows long.
  • Choose Flash attention (auto by default, or on/off) and provide Extra args if you already know you need additional llama-server options.

Only one Direct model runs at a time; selecting another stops the previous process first. While a model loads, Muse shows elapsed time rather than an estimated percentage. Open Log to see persistent chat messages and, for Direct, the server’s live process output—particularly useful when a model cannot load.

comfyui-muse Models

Muse does not include its own chat models. You choose a model available through LM Studio or Ollama, or point Direct at GGUF model files you have on disk. Choose according to the task:

Model typeWhen to use itExample
Text chat modelBrainstorm, expand, shorten, or rewrite prompts and tag lists.Ask for three variations of a product-photography prompt.
Vision-capable modelDiscuss an attached image or sampled video frames. For video, choose a model capable of understanding visual sequences, such as Qwen3-VL.Ask how the lighting changes across a clip.
Audio-capable modelDiscuss or transcribe attached audio through the Direct loader. Examples named by the author include Qwen2-Audio, Qwen2.5-Omni, Ultravox, and Voxtral.Ask about speech or sounds in a reference clip.

A model name alone is not a guarantee that every attachment will work. For Direct vision and audio models, keep the model’s .gguf file and its matching mmproj-*.gguf projector file in the same folder; Muse pairs them automatically and marks the model’s media capability in the menu.

Direct also recognizes split GGUF models by their first file and filters out diffusion-model GGUF checkpoints it encounters in ComfyUI model folders. A Flux or SDXL checkpoint may share the GGUF file format, but it is not a chat model.

What's New with comfyui-muse

V2 adds the Direct GGUF loader, so you can chat without LM Studio or Ollama, as well as audio attachments, editing and resending messages, and branching conversations. These changes make it easier to reuse an existing discussion, correct a request, or test a new creative direction without starting over.

For the author’s fuller account of the V2 changes, see RELEASE_NOTES.md. The provided documentation does not describe a version-by-version history beyond V2.

Troubleshooting comfyui-muse

If something is not working, start with the model, backend, and attachment type you selected. The Log panel is especially useful for Direct-loader problems.

ProblemWhat to try
No models appearConfirm the selected backend. For LM Studio or Ollama, check that the app is running and its address in Muse is correct, then press ⟳. For Direct, check that you added a folder containing chat-model .gguf files.
A Direct model will not loadOpen Log and read the load message. If memory is the issue, try Fit to GPU, fewer GPU layers, or a smaller context length. Muse may also retry automatically with less GPU offloading; the log shows the outcome.
Chat is unexpectedly slowCheck the Direct loader’s context length and GPU settings. An oversized context can exhaust VRAM or make generation much slower. The Log can help show whether the model fell back to fewer GPU layers or CPU operation.
An image or video is not understoodConfirm that you selected a vision-capable model. With Direct, check that its matching mmproj file is beside the model. For video, also check whether ComfyUI reports that OpenCV is missing.
Audio is ignored or a warning appearsUse the Direct backend with an audio-capable model and its matching projector file. LM Studio and Ollama audio-input support is limited in the APIs described by the author.
A reference file stops affecting repliesCheck that it is still in ComfyUI/input/ and selected under Guide materials for this chat. Muse flags missing files.
A reply stops before it finishesIncrease that chat’s Max reply tokens, then ask again. The panel indicates when a reply hits the set limit.
Rendering runs out of GPU memoryCheck that Unload on Run is enabled, or press Unload before rendering. To make room before chatting, enable Free ComfyUI VRAM.

Frequently asked questions

Does Muse send its chat text directly into my ComfyUI workflow?
No. Muse intentionally has no graph connections. Copy the text you like and paste it into your workflow.

Will my conversations still be there after a restart?
Yes. Muse saves chats and restores them across restarts.

Do I need LM Studio or Ollama to use Muse?
No. The Direct loader can run a compatible GGUF chat model from disk using llama-server.

Why does the model load again after I render?
With Unload on Run enabled, Muse unloads the chat model so ComfyUI can use the VRAM. It loads again when you send a new message.

Can I ask about a whole video?
You can attach a video, but Muse sends selected, timestamped frames—not every frame. Increasing the sampling rate or frame limit can provide more visual detail, subject to the model’s available context and your machine’s resources.

Learn More about comfyui-muse

Muse is released under the MIT license.

comfyui-muse Related Nodes

RunComfy
Copyright 2025 RunComfy. All Rights Reserved.

RunComfy is the premier ComfyUI platform, offering ComfyUI online environment and services, along with ComfyUI workflows featuring stunning visuals. RunComfy also provides AI Models, enabling artists to harness the latest AI tools to create incredible art.

comfyui-muse detailed guide | ComfyUI