cloud-run-gpu-image-update-quota-bypass

Cloud Run GPU Image Update - Quota Bypass

Problem

When deploying updated container images to an existing Cloud Run service with GPU (e.g., NVIDIA L4), gcloud run deploy fails with quota errors even though the service is already running with a GPU. The quota check blocks both zonal and non-zonal redundancy configurations, making it impossible to deploy updated code.

Context / Trigger Conditions

• gcloud run deploy returns:

  metadata.annotations[run.googleapis.com/maxScale]: You do not have quota for
  using GPUs with zonal redundancy.
  

  Followed by:

  metadata.annotations[run.googleapis.com/maxScale]: You do not have quota for
  using GPUs without zonal redundancy.
  

• The GPU service already exists and has a running revision
• You're trying to deploy an updated container image, not change GPU configuration
• The deploy script uses gcloud run deploy with --gpu flags

Solution

Instead of gcloud run deploy (which re-validates all resource quotas), use gcloud run services update which only updates the specified fields on the existing service:

# Instead of this (fails with quota error):
gcloud run deploy divine-transcoder \
  --image gcr.io/PROJECT/divine-transcoder \
  --region us-central1 \
  --gpu 1 --gpu-type nvidia-l4 \
  --cpu 4 --memory 16Gi \
  ...

# Use this (updates image on existing service):
gcloud run services update divine-transcoder \
  --region us-central1 \
  --image gcr.io/PROJECT/divine-transcoder:latest

Key differences:

• gcloud run deploy creates a new service or replaces the full configuration, triggering quota checks
• gcloud run services update --image only updates the container image on the existing service, preserving all existing GPU/CPU/memory configuration without re-validating quotas

Verification

# Verify new revision is active
gcloud run revisions list --service SERVICE_NAME --region REGION --limit=3

# Check the service is serving traffic
gcloud run services describe SERVICE_NAME --region REGION --format='value(status.url)'