video-understanding

video-understanding — Marlin-2B as eyes on a video

Two verbs, each on a single clip, each --json:

caption = what's in it (a scene description + a <start>–<end> event timeline). find = when one thing happens (a single start → end span).

Hard limits — read before using

1. One call sees ~2 minutes. Marlin samples ~2 fps up to ~240 frames, so a single caption/find only "sees" a bounded clip. On longer videos the timestamps drift and detail is lost.
2. Longer than ~2 min → window it. Cut overlapping windows with ffmpeg, run per window, then offset the returned timestamps by the window's start:

   ffmpeg -i long.mp4 -ss 0   -t 120 -c copy w0.mp4   # 0–120s
   ffmpeg -i long.mp4 -ss 110 -t 120 -c copy w1.mp4   # 110–230s (10s overlap)
   marlin caption w0.mp4 --json ; marlin caption w1.mp4 --json
   

   Every timestamp from w1 must have 110 added back.
3. find returns ONE span — there is no multi-find. It gives the single best match. To find every occurrence, window the video (as above) and run find per window, then collect the hits. Never assume one find call surfaces all matches.
4. Visual only. These modes don't read audio/speech. For "what was said," use a transcription tool.
5. No library search. Searching across a whole folder of videos is not in these verbs (that's the experimental index/search, not shipped — see the repo roadmap).

Resolution & memory (automatic)

Marlin's per-frame budget is ~200K pixels (≈448², 2 fps). The CLI auto-downscales the clip to that budget before inference — mirroring the model's own smart_resize — so a 1080p/4K clip is decoded small: much faster and far less memory, with no accuracy loss (the model would resize to this anyway, just after a costly full-res decode that can OOM a laptop). You don't have to do anything; it's on by default.