generating-video-clips

Generating video clips (still to clip, then lip-sync)

Overview

A finished still becomes a 4-9 s clip with an image-to-video model. Then, if a character talks on screen, a lip-sync model re-times their mouth to your own dialogue. You cannot watch the result: judge every clip from extracted frames (scripts/take_sheet.py, and the viewing-videos skill), and let the human have the final say on motion.

Requirements: Python 3 (stdlib only) and ffmpeg/ffprobe. Keys go in env vars: GEMINI_API_KEY (a Google AI Studio key; an image-scoped one works, and GEMINI_IMAGE_API_KEY is read as a fallback) for Omni, and FAL_KEY (the full key exactly as fal shows it) for lip-sync. Never print them.

Choose the model

Model	Reached through	Use for	Notes
Gemini Omni Flash 1.1 (gemini-omni-1.1-flash)	Google Interactions API, AI Studio key	Default for illustrated, anime and cartoon stills; also worked on a photoreal still (one test)	keeps art style and faces; any input aspect (a 2:3 still came back as clean 9:16, so no need to outpaint or crop first); 720p 9:16 or 16:9; synchronous, ~25-35 s per clip; adds its own audio (mute it)
Veo 3.1 / Veo 3.1 Fast	Gemini API, image as the start frame	photoreal stills	not used in our runs; check current docs first
Seedance 2.5	ElevenLabs Flows, or ByteDance directly	alternative look	needed account approval when tried
Any model inside ElevenLabs Flows	ElevenLabs API	one key for everything	API access needs the Pro plan (402 paid_plan_required) and the key's image_video_generation permission (401 without it); Omni there is app-only

For lip-sync, Sync 3 on fal (fal-ai/sync-lipsync/v3) is the default. It works video-to-video, so it keeps the clip's motion and re-times only the mouth. Comparison and prices: references/providers.md.

Animate a still

python3 <this-skill-dir>/scripts/omni_generate.py jobs.json clips/ [--takes 3] [--only S05]

{"base": "Bring this still to life as it is: same art style, palette, light and layout, and every character keeps the face, hair, build and clothes they have in it. Natural, fluid movement. Nothing written on screen, no speech bubbles, nobody new, one continuous shot. ",
 "jobs": {"S05": {"image": "panels/p05.png",
                  "prompt": "[0-3s] The woman in the red jacket lifts the phone and talks, excited. [3-6s] The friend in the green hoodie leans back on the sofa and laughs. Slow camera push-in."}}}

How to write the prompt (full guide with examples in references/gemini-omni.md):

• A per-second timeline, one beat per bracket. The clip runs about as long as the timeline.
• Name people by what they wear, never by name: the model doesn't know who "Dana" is.
• Say "talks" for anyone who will be lip-synced, so the mouth already moves.
• Pin what must not change: "the camera stays still", "one single window on the left wall", "exactly ONE hooded figure", "her hands stay flat on the table". Each pin in references/failure-modes.md came from a real failure.