bytedance/omnihuman/v1.5

What makes Omnihuman 1.5 stand out

Omnihuman 1.5 is a high-fidelity image-to-video and audio-to-video model that converts a single portrait and a voice track into realistic, lip-synced avatar performance. Built by ByteDance (Omnihuman Lab), it fuses a Multimodal LLM with a Diffusion Transformer to preserve facial structure, identity, and continuity while reacting to audio semantics. Components such as the “Pseudo Last Frame” and shared attention stabilize character features across frames and align motion with rhythm, prosody, and intent. Omnihuman 1.5 (also referred to as Omnihuman v1.5) adapts to narrative, dialogue, and singing, and can follow optional text guidance for camera and gesture control without sacrificing realism. Key capabilities:

• Single-image + audio avatar synthesis with natural lip-sync and expressive micro-motions.
• Audio-driven gestures: interprets rhythm, prosody, and meaning for context-aware motion.
• Optional text prompts to guide camera moves, gestures, and emotional tone.
• Identity stability via Pseudo Last Frame and shared attention for temporal coherence.
• Multi-person scenes and dialogue with consistent characters over longer clips.
• Mask-based subject targeting to control who speaks in complex images.
• Deterministic seeding for reproducible results and efficient iteration.

Prompting guide for Omnihuman 1.5

Start by providing a clear portrait image_url and an audio_url under 35 seconds; add an optional prompt to specify camera movement, gesture emphasis, or emotion. Use mask_url to mark the speaking subject if multiple faces are present. Keep guidance concise and unambiguous (e.g., preserve identity, subtle head motion). Omnihuman 1.5 interprets audio semantics first, then applies prompt-level refinements, yielding coherent performance that aligns lip, gaze, and gesture with the track. Prompts can be written in Chinese, English, Japanese, Korean, Mexican Spanish, or Indonesian. Examples:

• “Neutral studio framing; preserve identity; minimal background motion.”
• “Subtle nods and hand cues on emphatic words; soft push-in camera.”
• “Cheerful tone, upbeat gestures; keep head centered; do not alter hair.”
• “Storytelling pace with natural pauses; eye contact; gentle left-to-right pan.”
• “Singing performance; open-mouth vowels emphasized; hands at chest level.” Pro tips:
• Prioritize clean, well-lit portraits; crop distractions and frontalize the face.
• Keep audio high quality and under 35s; remove silence and background noise.
• Use mask_url for multi-person images to fix the speaking subject.
• State constraints explicitly: what to preserve vs. what to change (camera, gestures).
• Iterate with seed control for reproducibility; refine with short, focused prompt edits.