data-storage

Data Storage

Choose where records live, how they are keyed and queried, and how the store grows past a single machine. Storage is the hardest layer to change later: a wrong data model or shard key calcifies into a scaling ceiling, and getting replication wrong silently serves stale or lost data.

When to reach for this

Any system that persists state: picking SQL vs NoSQL, designing a schema and its access paths, adding indexes, splitting a hot table, distributing data across nodes (sharding/partitioning), adding read replicas, or deciding what to denormalize. Reach here the moment "store the data" needs a concrete key and query shape.

When NOT to

Don't shard, add replicas, or reach for NoSQL before a number forces it (YAGNI). A single well-indexed relational node handles ~1k QPS and tens of GB to low TB comfortably — most systems never outgrow it. Sharding multiplies operational cost and breaks joins/transactions; add it only when one node's write throughput or dataset size is genuinely exceeded (→ back-of-the-envelope). Caching reads (→ caching) and adding read replicas are cheaper first moves than sharding.

Clarify first

Answer these before choosing a store or topology — they decide the design:

• Data shape & relationships — flat key-value? rich relations needing joins? document blobs? a graph of connections? Drives SQL vs NoSQL.
• Access patterns — how is data read and written, not just where it lives. Point lookups by key, range scans, ad-hoc queries, aggregations? Model the store around the queries it must serve.
• Read:write ratio & scale — QPS each way, total size now and at retention. (→ back-of-the-envelope for QPS, storage, and shard counts.)
• Consistency need — must reads see the latest write, or is eventual OK? Are multi-record transactions required? (CAP/consistency theory → consistency-coordination.)
• Latency & durability targets — p99 read/write budget, and how much recent data the system can afford to lose on a node failure.