analyze-footage

Analyze Footage — AI Video Understanding

You are helping the user understand their existing video files using TwelveLabs MCP. This gives your agent "eyes" to see what's in video footage.

Load the remotion-production skill for the video-analysis rule.

Workflow

1. Identify Video Files

Ask the user which video files to analyze, or look for them:

# Check common locations
ls public/footage/ public/raw/ public/video/ public/ 2>/dev/null | grep -E '\.(mp4|mov|avi|mkv|webm)$'

If no files found, ask the user where their footage is located.

2. Upload to TwelveLabs

For each video file, upload it as an asset (no index needed for analysis):

create-asset:
  file: [absolute path, e.g. /path/to/project/public/footage/source-video.mp4]

Keep the returned asset ID. If the next call says the asset isn't ready yet, wait a few seconds and retry.

3. Analyze Content

Scene Breakdown — a timestamped scene map, as the video-analysis rule shows:

async-analyse-video:
  assetId: [asset ID]
  modelName: "pegasus1.5"
  analysisMode: "time_based_metadata"
  segmentDefinitions: [one "shots" definition with fields: description, shot_type, people, on_screen_text, usable]

Poll get-analyse-task every 5-10 seconds until it's ready, then parse result.data (a JSON string keyed by the segment definition id). Videos up to 2 hours work.

For a direct question ("which shots show the product?", "pick clips for a 30s promo"), use sync-analyse-video (assetId, modelName: "pegasus1.5", prompt, and a jsonSchema so you get start/end times back as numbers). Sync analysis is capped at 1 hour of video.

Always pass modelName: "pegasus1.5": the server's default model has been removed from the API.

Present as:

📹 Video Analysis: [filename]

Scenes:
  0:00 - 0:15  │ Logo animation, dark background
  0:15 - 0:45  │ Speaker at podium, introducing topic, conference setting
  0:45 - 1:20  │ Product demo on laptop screen, close-up
  1:20 - 1:45  │ Audience reaction, medium shot, clapping
  1:45 - 2:00  │ Call to action slide, company branding

Key Elements (from the segment fields):
  - People: 1 speaker, audience (~30 people)
  - Shot types: wide, medium, close-up, screen capture
  - Text on screen: company logo, product name
  - Setting: conference room, professional lighting