metadata-storage-source-of-truth-desync

Metadata-Storage Source of Truth Desync

Problem

When a system has both metadata (KV store, database status field) and actual content in object storage, the metadata can become stale. If the serving handler gates on metadata status before checking storage, content that exists becomes inaccessible.

Common causes:

• Webhook callbacks (transcode complete, migration done) failed silently
• Backfill scripts created content but didn't update metadata
• Race conditions between metadata write and storage write
• Manual operations that bypassed the normal status update flow

Context / Trigger Conditions

• API returns "processing" or "202 Accepted" for content that actually exists in storage
• transcode_status is Processing/Pending/None but HLS files exist in GCS
• Content works when accessed directly via storage URL but not through the API
• Backfill script ran successfully (content in bucket) but metadata wasn't updated
• Webhook endpoint had downtime or returned errors during batch processing

Solution

Pattern: Storage-First with Metadata Auto-Repair

Check storage (source of truth) BEFORE checking metadata status. If content exists, serve it and fix metadata as a side effect.

// BAD: Metadata-first (blocks access when metadata is stale)
match metadata.transcode_status {
    Some(Complete) => { /* serve from storage */ }
    Some(Processing) => { return 202; }  // Blocks even if content exists!
    _ => { trigger_transcoding(); return 202; }
}

// GOOD: Storage-first with auto-repair
match download_from_storage(&path) {
    Ok(content) => {
        // Content exists — serve it and fix metadata if needed
        if metadata.transcode_status != Some(Complete) {
            eprintln!("[FIX] {} has content but status was {:?}", id, metadata.transcode_status);
            update_status(id, Complete);  // Auto-repair
        }
        serve(content)
    }
    Err(NotFound) => {
        // Content truly doesn't exist — now check metadata for status
        match metadata.transcode_status {
            Some(Processing) => return 202,
            _ => { trigger_processing(); return 202; }
        }
    }
}