service-connectivity

Service Connectivity

Every call one service makes to another is a bet that the other side is up, fast, and honest about its identity. Distributed systems fail at the seams between services far more often than inside any single one — a slow dependency without a timeout, a retry storm without backoff, or a connection that was never actually authenticated all turn a local problem into a cascading one.

Design connectivity assuming the other side will eventually be slow, wrong, or absent — because over enough time, it will be. A call without a timeout is a promise to wait forever, and nothing downstream should ever make that promise.

1. Make service discovery answer "is this instance actually healthy," not just "does it exist"

Discovery mechanisms (DNS-based, a registry, or a mesh control plane) that only track which instances exist — without incorporating readiness — will happily hand out an address for an instance that's up but not ready to serve. Discovery and health need to be the same signal, or a caller ends up needing its own separate health check on top, which most callers skip.

• Prefer discovery that's health-aware by construction (e.g., a mesh or orchestrator that removes not-ready instances from the result set) over a static list plus a hope.
• In Kubernetes, this is exactly the Service/Endpoints readiness relationship — see kubernetes-networking for that mechanism specifically.

Done when: a caller resolving a dependency only ever gets instances that are actually ready to serve.

2. Set timeouts before retries, and retries before circuit breakers

These three exist in a dependency order: a timeout bounds how long one call can hang, a retry policy decides what to do about the failures a timeout produces, and a circuit breaker stops retrying a dependency that's clearly down instead of hammering it. Configuring retries without a timeout, or a circuit breaker without sane retries underneath, just moves the failure mode instead of fixing it.