istio-envoy-service-mesh-mechanics

Istio and Envoy: Data-Plane Mechanics

Purpose

Most mesh problems (traffic bypassing the proxy, 503s with UF/NR flags, mTLS failures, latency) are explained by how packets get redirected and how Envoy's configuration was pushed. This skill covers both sidecar and ambient data planes and the commands to inspect them.

Agent Protocol

Trigger

"istio internals", "envoy xds", "sidecar injection", "iptables redirect", "ambient mode", "ztunnel", "waypoint", "istio-cni", "proxy-config", "mesh overhead".

Input Context

• Istio version and data-plane mode (sidecar or ambient), CNI in use.
• Failing request path (source workload → destination service, port, protocol).
• Envoy response flags / access logs.

Output Artifact

Explanation + diagnostic commands + config fix. No file unless requested.

Response Format

Mode: {sidecar|ambient (ztunnel only)|ambient + waypoint}
Path: {app -> redirect -> proxy -> upstream}
Finding: {listener/route/cluster/endpoint/cert issue}
Commands: {istioctl ...}
Fix: {resource change}

Completion Criteria

• Traffic path identified for the specific mode.
• Root cause tied to a concrete Envoy config object or redirect rule.

Control Plane → Envoy (xDS)

Istiod watches Kubernetes (Services, EndpointSlices, Istio CRDs) and pushes config to proxies over a gRPC xDS stream (Aggregated Discovery Service, delta xDS by default in recent releases):

xDS	Object	Istio source
LDS	Listeners (ports, filter chains)	Services, Sidecar resource, Gateways
RDS	HTTP route tables	VirtualService / HTTPRoute
CDS	Upstream clusters (LB, TLS, circuit breaking)	DestinationRule
EDS	Endpoints (pod IPs, locality, health)	EndpointSlices
SDS	Certificates/keys	Istiod CA (SPIFFE identities)

Envoy runs a main thread (xDS, admin) plus worker threads, each running its own event loop over accepted connections; config changes are applied without restarting.

Sidecar Mode Interception

• Injection adds istio-proxy (and historically istio-init, which needs NET_ADMIN). With istio-cni, the CNI plugin programs redirection instead, so app pods need no elevated init container.
• iptables nat rules (chains ISTIO_INBOUND, ISTIO_REDIRECT, ISTIO_OUTPUT):
  • outbound app traffic → Envoy port 15001 (virtualOutbound listener),
  • inbound traffic → 15006 (virtualInbound listener),
  • traffic from the proxy's own UID (1337) is excluded to avoid loops.
• Other ports: 15000 admin, 15020 merged metrics/health, 15021 health, 15090 Envoy Prometheus stats.
• Exclusions: traffic.sidecar.istio.io/excludeOutboundPorts, excludeOutboundIPRanges annotations.
• Startup race: app sends traffic before Envoy is ready → set holdApplicationUntilProxyStarts: true (or native sidecar mode, ENABLE_NATIVE_SIDECARS).