gemini-video-understanding

Gemini Video Understanding Skill

Purpose

This skill enables video understanding workflows using the Google Gemini API, including video summarization, question answering, transcription with optional visual descriptions, timestamp-based queries (MM:SS), scene/timeline detection, video clipping, custom FPS sampling, multi-video comparison, and YouTube URL analysis.

When to Use

• Summarizing a video into key points or chapters
• Answering questions about what happens at specific timestamps (MM:SS)
• Producing a transcript (optionally with visual context) and speaker labels
• Detecting scene changes or building a timeline of events
• Analyzing long videos by clipping to relevant segments or reducing FPS
• Comparing multiple videos (up to 10 videos on Gemini 2.5+)
• Analyzing public YouTube videos directly via URL

Required Libraries

The following Python libraries are required:

from google import genai
from google.genai import types
import os
import time

Input Requirements

• File formats: MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP
• Size constraints:
  • Use inline bytes for small files (rule of thumb: <20MB).
  • Use the File API upload flow for larger videos (most real videos).
• YouTube:
  • Video must be public (not private/unlisted) and not age-restricted.
  • Provide a valid YouTube URL.
• Duration / context window (model-dependent):
  • 2M-token models: ~2 hours (default resolution) or ~6 hours (low-res).
  • 1M-token models: ~1 hour (default) or ~3 hours (low-res).
• Timestamps: Use MM:SS (e.g., 01:15) when requesting time-based answers.

Output Schema

All extracted/derived content should be returned as valid JSON conforming to this schema:

{
  "success": true,
  "source": {
    "type": "file|youtube",
    "id": "video.mp4|VIDEO_ID_OR_URL",
    "model": "gemini-2.5-flash"
  },
  "summary": "Concise summary of the video...",
  "transcript": {
    "available": true,
    "text": "Full transcript text (may include speaker labels)...",
    "includes_visual_descriptions": true
  },
  "events": [
    {
      "timestamp": "MM:SS",
      "description": "What happens at this time",
      "category": "scene_change|key_point|action|other"
    }
  ],
  "warnings": [
    "Optional warnings about limitations, missing timestamps, or low confidence areas"
  ]
}