microservice-api-endpoint-routing-404

Microservice API Endpoint Routing 404 Debugging

Problem

API endpoint returns 404 even though the code implementing it has been deployed. This commonly happens in microservice architectures where multiple services share the same hostname and HTTPRoutes/Ingress rules determine which service handles which paths.

Context / Trigger Conditions

• New endpoint returns 404 but code is definitely deployed
• Multiple services (e.g., relay + api) share the same external hostname
• Service logs show "Health server listening" or similar health-only messages
• HTTPRoute or Ingress rules split traffic by path prefix
• Monorepo builds multiple Docker images from same commit

Solution

Step 1: Identify which service actually handles the path

Check HTTPRoute/Ingress rules to see routing:

kubectl get httproute <name> -n <namespace> -o yaml | grep -A 50 "rules:"

Look for path matching rules and which backend service they point to:

• /api/* might go to api-service:8080
• / might go to websocket-service:7777

Step 2: Test the endpoint directly on each service from inside cluster

kubectl run curl-test -n <namespace> --image=curlimages/curl --restart=Never \
  -- curl -v "http://<service-name>:<port>/<endpoint>" \
  && sleep 5 \
  && kubectl logs -n <namespace> curl-test \
  && kubectl delete pod -n <namespace> curl-test

This bypasses external routing and tests directly against each service.

Step 3: Check service logs for endpoint registration

kubectl logs -l app=<service-name> -n <namespace> --tail=50 | grep -i -E "(endpoint|route|api|listening)"

Look for messages like:

• "Health server listening on 0.0.0.0:8080" = health only, no API
• "API server listening" or "Registered /api/..." = serves API endpoints

Step 4: For monorepos, check if matching image exists for correct service

If feature was added to service A but endpoint is served by service B, check if service B has the same commit tag:

gcloud artifacts docker images list <registry>/<service-b> --include-tags --limit=10