multi-cluster-api-data-mismatch

Multi-Cluster API Data Mismatch

Problem

When investigating why an API returns data that doesn't match direct database queries, the root cause may be that your kubectl context points to a different cluster (e.g., POC) than the one actually serving the public API (e.g., production). Both clusters have the same schemas and similar data volumes, making the mismatch non-obvious.

Context / Trigger Conditions

• API returns records that direct database queries (kubectl exec ... clickhouse-client) can't find
• Data counts are similar (e.g., both have ~20k records) but specific records differ
• You're debugging a data pipeline that "should work" but labels/counts never update
• Multiple kubectl contexts exist (production, staging, POC, test)
• The API response header shows server: nginx but no ingress exists in the current cluster's namespace

Solution

Step 1: Verify which cluster serves the domain

# Check DNS for the public API domain
dig relay.divine.video +short
# → 34.58.27.79

# Check your current cluster's gateway/ingress IP
kubectl get gateway -A  # or kubectl get ingress -A
# → 35.184.10.63  (different IP = different cluster!)

Step 2: Check all available contexts

kubectl config get-contexts
# Look for production vs staging vs POC contexts

Step 3: Switch to the correct cluster

kubectl config use-context <production-context-name>

Step 4: Verify the database connection

# Check what database the API actually connects to
kubectl get secret -n <namespace> <db-credentials> -o jsonpath='{.data.DATABASE_URL}' | base64 -d
# Production might use managed services (ClickHouse Cloud, Cloud SQL)
# POC might use in-cluster pods

Step 5: Also check for ExternalSecrets when updating secrets

If secrets are managed by ExternalSecrets operator:

# Check if ExternalSecrets manages the secret
kubectl get externalsecret -n <namespace>

# If yes, update the SOURCE (e.g., GCP Secret Manager), not the k8s secret
gcloud secrets versions add <secret-name> --project=<project> --data-file=-

# Force sync after updating
kubectl annotate externalsecret <name> -n <namespace> force-sync=$(date +%s) --overwrite

# Verify the k8s secret updated
kubectl get secret <name> -n <namespace> -o jsonpath='{.data.<key>}' | base64 -d