engineering-backend-architect

Backend Architecture Guide

Overview

This guide covers scalable backend system design, database architecture, API development, and cloud infrastructure patterns. Use it when making decisions about data schemas, service boundaries, caching strategies, security architecture, or performance optimization.

Architecture Decision Rules

System Design

• When choosing between microservices and a monolith, start with a modular monolith unless the team already operates multiple services in production -- microservices add deployment and observability cost that slows small teams.
• When designing database schemas, add partial indexes on high-cardinality columns filtered by common WHERE clauses (e.g., WHERE is_active = true) because full-table indexes waste I/O on rows that queries never touch.
• When versioning APIs, use URL-prefix versioning (/v1/, /v2/) for public APIs and header versioning for internal APIs because URL prefixes are easier for external consumers to discover and cache.
• When building event-driven systems, ensure every event includes a unique idempotency key and a schema version field so consumers can safely retry and handle schema evolution.

Reliability

• When a downstream service is unreliable, wrap calls in a circuit breaker (e.g., opossum for Node.js) -- open after 5 consecutive failures, half-open after 30 seconds, close after 3 successes.
• When designing backup strategies, combine continuous WAL archiving with daily base backups and test restores weekly against a staging database to verify RTO/RPO targets.
• When implementing health checks, expose /health/live (process is running) and /health/ready (dependencies are reachable) as separate endpoints because Kubernetes liveness and readiness probes serve different purposes.

Performance

• When Redis is used for caching, set TTLs explicitly on every key and use cache-aside (lazy loading) rather than write-through unless write latency is more important than read consistency.
• When processing large datasets, use cursor-based pagination instead of OFFSET/LIMIT because OFFSET scans and discards rows, degrading linearly with page depth.
• When designing a new service, ensure it is stateless so any instance can handle any request; store session data in Redis or a database so horizontal scaling requires only adding instances behind the load balancer.
• When adding a new query path, run EXPLAIN ANALYZE before merging and reject any query that performs a sequential scan on a table with more than 10k rows -- add an index or rewrite the query.
• When introducing a cache layer, define an explicit invalidation strategy (TTL, event-driven purge, or versioned keys) in the design doc before implementation to prevent stale reads.