video-understand

Video Understand

Build the shared evidence layer once. Keep observations in source time and leave editorial decisions to downstream skills.

This skill is a prerequisite for /video-cut, /video-to-shorts, /video-add-captions, /video-add-content-cards, and /video-add-graphic-motion. Run it first so those skills consume the same validated evidence and timeline.

Dependencies

Require ffmpeg/ffprobe, Python, and faster-whisper for transcription. Check them before processing media.

Workflow

1. Initialize a project from the original source:

   python scripts/init_project.py path/to/source.mp4 path/to/my-video-project
   

   This creates input/, review/00-video-understanding/, final/, the minimal machine-facing work/ tree, an identity timeline, project.json, media facts, and START-HERE.md. It does not create folders for unselected edit operations.

2. Probe again only when the source needs an explicit metadata refresh:

   python scripts/probe.py input/original-video.mp4 work/understand/media.json
   

3. Extract 16 kHz mono audio and transcribe it:

   ffmpeg -y -i input/original-video.mp4 -ac 1 -ar 16000 work/cache/audio16k.wav
   python scripts/transcribe.py work/cache/audio16k.wav work/understand/transcript medium `
     --lang auto --cache-dir work/cache/faster-whisper
   

   Use --lang auto for unknown or mixed-language speech. Never infer the spoken language from the language of the user's prompt. Pass a fixed language such as --lang zh only when the audio itself or explicit user metadata establishes it. Keep model downloads in the project-local work/cache/faster-whisper/ cache. Faster-whisper may emit an occasional point-timed word with equal start/end values; the shared timeline mapper preserves it as a 1 ms interval so captions and derivatives do not silently lose text.

4. Generate objective metrics and semantic candidates:

   python scripts/analyze.py work/understand/transcript.json work/understand/analysis.json