drawing-video

drawing-video

Footage in, narrated comic strip out. Five stages: probe → ground → narrate → draw → QA/compose. The hard part is not the drawing; it is not lying about what the footage shows. One stage is a gate.

Requires: processing-video (ffmpeg, present), invoking-gemini (image + video models), proxy.env (CF gateway creds — auto-read by the gemini client).

---

Stage 0 — Probe & extract frames

ffprobe -v quiet -show_entries format=duration -of csv=p=0 in.mov
mkdir -p frames
ffmpeg -v error -i in.mov -vf "fps=1,scale=640:-1" frames/f_%03d.png   # 1/sec, downscaled

Keep full-resolution stills too — you will feed them to the image model as reference. For a subject hidden in clutter, pull a few full-res frames at specific timestamps: ffmpeg -v error -ss 11 -i in.mov -frames:v 1 hr.png.

---

Stage 1 — GROUND THE STORYLINE (the gate)

Never narrate a scene whose subjects and action you have not confirmed from the footage. Thumbnail-sized frames read poorly: one pass built an entire "indoor cyclist in a garage" storyline for a video that was actually a leashed dog watching a deer — no bicycle in any frame, the leash visible in four — then fed the fiction to the image model, which dutifully drew it. A wrong storyline launders a hallucination into finished art. The absence of a clear read is not license to invent a vivid one.

Two ways to ground. Prefer Mode B — it is the more reliable and reads the whole clip, not sampled stills.

Mode B (recommended): native video understanding via gemini-3.5-flash

gemini-3.5-flash parses video directly. Compress first, send as inline data.

ffmpeg -v error -i in.mov -vf "scale=480:-2,fps=8" -c:v libx264 -crf 30 -preset veryfast -an small.mp4

import sys, base64; sys.path.append('/mnt/skills/user/invoking-gemini/scripts')
from gemini_client import get_cf_credentials, _cf_request
from pathlib import Path
c = get_cf_credentials()
b = base64.b64encode(Path("small.mp4").read_bytes()).decode()
prompt = ("Watch this video and describe the actual storyline in 3-4 sentences: which "
          "subjects appear, what happens, the arc. Be literal — only what is visibly present.")
contents = [{"parts": [{"inlineData": {"mimeType": "video/mp4", "data": b}}, {"text": prompt}]}]
resp = _cf_request("gemini-3.5-flash", contents,
                   {"temperature": 0.3, "thinkingConfig": {"thinkingLevel": "minimal"},
                    "maxOutputTokens": 600}, c)
print("".join(p.get("text","") for p in resp["candidates"][0]["content"]["parts"]))