viewing-videos

Viewing Videos

Overview

Claude can't watch video, but it reads images well. Turn the video into the fewest frames that answer the question with ffmpeg, triage them on a contact sheet, then Read the interesting ones at full resolution. Requires ffmpeg + ImageMagick montage (both via Homebrew).

Step 0: Probe first

ffprobe -v error -show_entries format=duration:stream=codec_type,width,height -of default=noprint_wrappers=1 video.mp4

Duration picks the strategy; resolution sets expectations — 1080p+ screen content reads fine, at 720p big KPI numbers are fine but dense tables/spreadsheets need the crop+upscale step below.

Pick a strategy

Situation	Strategy
You know the moment (timestamped transcript, user says "at 12:34")	Targeted single frames
Screen shares / slides / UI demos, unknown moments	Scene-change sweep (default)
Continuous motion (camera footage, gameplay) — scenes never "change"	Interval sampling (see frame budget below)

Targeted frame

-ss before -i = fast seek:

ffmpeg -ss 12:34 -i video.mp4 -frames:v 1 /tmp/frames/t12m34s.png

Scene-change sweep → timestamp names → contact sheet

rm -rf /tmp/frames && mkdir -p /tmp/frames
ffmpeg -hide_banner -i video.mp4 -vf "select='gt(scene,0.2)',showinfo" -fps_mode vfr /tmp/frames/frame_%03d.png 2> /tmp/frames/log.txt
cd /tmp/frames && grep -o 'pts_time:[0-9.]*' log.txt | cut -d: -f2 | awk '{n++; t=int($1); printf "mv frame_%03d.png t%02dm%02ds.png\n", n, t/60, t%60}' | sh
montage -label '%t' -font /System/Library/Fonts/Helvetica.ttc t*.png -tile 6x -geometry 320x180+4+4 -pointsize 18 contact_%d.png

Read contact_0.png to triage (timestamps are printed under each tile), then Read only the frames that matter at full resolution. Threshold 0.2 works for meetings: webcam segments cluster harmlessly, screen shares fire exactly on slide/page changes. Footage with constant motion → hundreds of frames; switch to interval sampling.