System design 101

System design looks like it is about boxes and arrows. It is really about trade-offs. Every big decision in a large system is a purchase: you buy lower latency with memory, you buy availability by giving up some consistency, you buy write throughput by taking on operational pain. The boxes and arrows are just the receipt.

That is why there is no single "correct" design for a URL shortener or a chat app — only designs that make sense under stated constraints. Change the constraints and the right answer changes too. A feed for ten thousand users is one SQL query. The same feed for ten million users is a pipeline with three kinds of storage. Neither is wrong; they answer different questions.

This article is the foundation I think every engineer should be able to reason from: the numbers that make estimation possible, the standard building blocks, and — most important — when each block is worth its cost.

Start with the numbers

Most intuition about large systems is really arithmetic. Keep a handful of latencies roughly in your head and half of your design questions answer themselves — because most architecture exists to avoid the slow rows of this table.

The exact values change with every hardware generation. The ratios are what you keep:

• Memory is about a thousand times closer than another continent.
• Reading data in order beats jumping around by orders of magnitude.
• One cross-region call spends the whole latency budget of an interactive request — chatty work must stay inside one region.

For sizing, powers of two are the units of the trade:

Estimation is where the numbers pay off. Take a photo-sharing app with ten million daily users, each uploading about one photo every ten days at 2 MB:

writes:   10M / 10 = 1M photos/day  ≈ 12/s average, ~40/s at peak
storage:  1M × 2 MB = 2 TB/day      ≈ 730 TB/year, ×3 replicas ≈ 2 PB
reads:    ~100 views per photo      → reads outnumber writes 100:1
          — design the read path first

Thirty seconds of arithmetic just made the first real decision: this system is a CDN and a cache with a database behind it, not a database with a cache in front. That is what the estimate is for. Not precision — direction.

What scaling actually means

Two pairs of words get mixed up constantly, and keeping them straight is half of talking clearly about systems.

Performance — how fast the system is for one user. Scalability — whether it stays fast as users, data, and traffic grow.

A performance problem is slow for a single user, and you reach for a profiler. A scalability problem is fast for one user and slow for ten thousand, and you reach for architecture. Calling one by the other's name wastes months.

Latency — how long one request takes. Throughput — how many requests finish per second.

They trade against each other: batching writes raises throughput but makes each write slower. And when you measure latency, ask for the p99 — the time the slowest 1% of requests see — because averages hide exactly the users having the worst day.

Scaling itself goes in two directions:

• Vertical (scale up) — buy a bigger machine. No code changes, and the right first move more often than pride admits. But it has a hard ceiling, the price grows faster than the power, and it does nothing when that one machine dies.
• Horizontal (scale out) — add more machines. No practical ceiling and it survives failures, but it forces the question that shapes everything after it: where does state live? The moment two servers handle the same user, neither can keep anything important in its own memory.

Staying available

Availability — the fraction of time the system answers correctly, usually written in "nines."

Two lessons hide in the table. Each extra nine costs roughly ten times more effort than the last, so "five nines" is a promise about your on-call rotation, not just your architecture. And availability multiplies:

• Two services at 99.9% chained one after the other: 99.8% — worse than either alone, because either failing fails the pair.
• Two redundant paths side by side: 99.9999% — both must fail at once.

Redundancy is the only trick that adds nines instead of eating them. The standard shapes of it:

• Active–passive failover — a standby machine takes over when the active one dies (a heartbeat between them decides). Simple; you pay for hardware that does nothing until the worst day, and failover takes seconds to minutes.
• Active–active — both machines serve traffic all the time; losing one just shrinks capacity. Better utilization, but both must handle writes or traffic routing gets clever.
• Replication — the data-layer version of the same idea, covered with databases below.

Failover has its own classic bug: if the heartbeat link fails but both machines are healthy, each may decide it is the leader — split brain. It is why serious systems use quorum-based leader election instead of a single wire; Raft and Paxos are the standard recipes.

The shape of a scalable application

Almost every large web system ends up with the same skeleton. It is worth knowing cold — not to copy it from memory, but because every specialized design is this skeleton with one part made bigger.

Walk the request through it. DNS turns the name into an address. Static files — images, scripts, video — never reach your servers at all; the CDN serves them from a location near the user. Dynamic requests hit a load balancer, which spreads them across app servers that keep no state of their own: sessions live in Redis, files in object storage, records in the database. That statelessness is the whole trick — any server can handle any request, so adding capacity means adding machines, and losing a machine loses nothing.

Below the app tier, the data layer splits by job: a cache for hot reads, a database for the truth, a queue for work that should not keep the user waiting, object storage for big files (S3-style: write once, read by URL, cheap and effectively bottomless). The rest of this article is really this diagram, one box at a time.

DNS and CDNs

DNS — the distributed directory that turns a domain name into an IP address.

A lookup walks a chain — resolver, root, TLD, authoritative server — and survives only because every level caches answers, controlled by a TTL. That TTL is a real design lever: a low one lets you move traffic within minutes at the cost of more lookups; a high one means a bad record sticks around. Managed DNS can also answer based on where the user is (geo or latency routing), which makes DNS the outermost load balancer you own.

CDN — a network of servers near users that serves your content so requests never travel to your origin.

• Pull CDN — fetches a file from your origin the first time someone asks, then caches it at the edge. Almost no setup, ideal for lots of content; the first visitor per region pays the slow path.
• Push CDN — serves only what you upload to it. Full control, right for a small set of heavy files you know in advance.

The extreme of taking this seriously is Netflix: they built Open Connect, storage boxes that sit physically inside ISP networks, so most Netflix traffic never crosses the public internet at all. The lesson travels well — the cheapest request is the one that never reaches you.

Load balancing

Load balancer — a server that spreads incoming requests across many machines, hides their failures, and usually terminates TLS.

The spreading is the visible job. The important jobs are the other two: health checks pull a dead server out of rotation within seconds, which turns "a server died" from an outage into a non-event.

Balancers work at two levels:

• Layer 4 — routes on IP address and port. Fast, cheap, blind to what the request says.
• Layer 7 — reads the request itself, so it can send /api and /static to different pools, split off traffic to test a new release, or keep a WebSocket pinned to one server. The sensible default for web work.

Common algorithms, in the order you should reach for them:

• Round robin — take turns. Fine when servers and requests are similar.
• Least connections — send to the least busy. Better when request cost varies.
• Hash on a key — same client always lands on the same server. For when locality matters (sticky sessions, per-user caches).

Remember the balancer is itself a single point of failure: production runs at least a pair with failover, or uses the cloud provider's managed one — which is a pair someone else operates.

Reverse proxy — a server that sits in front of your backends and handles TLS, compression, caching, static files, and rate limiting.

A reverse proxy is useful even with a single backend. A load balancer is just a reverse proxy whose reason to exist is having several.

The application layer

Monolith — one deployable unit containing all the features. Microservices — many small services, each owning one capability and its own data, talking over the network.

The honest ordering: start with a monolith. One codebase, one deploy, function calls instead of network calls, one database transaction around a whole operation. Microservices buy you independent scaling and independent deploys for many teams, and they charge for it in the currency this whole article is about — every function call becomes a network call with latency, timeouts, retries, and partial failure. The rough rule: split when teams block each other, not when the code gets big.

Two pieces of machinery come with the split:

• API gateway — the single front door: routing, authentication, rate limiting, and one stable public API while services change behind it.
• Service discovery — how services find each other when instances come and go (Consul, etcd, or the platform's DNS). Registered instances are health-checked, so callers only see live ones.

The database ladder

Databases scale through a sequence of increasingly expensive moves. The discipline is taking them in order: each rung fixes a real limit and adds a real cost, and skipping ahead buys complexity before it buys you anything.

Rung zero: tune what you have. Before any new boxes:

• Add indexes for your real queries — and know each index slows writes a little.
• Find slow queries and fix them (every database will log them).
• Cap result sizes, avoid N+1 query patterns, use connection pooling.

This rung is unglamorous and regularly buys a 10× before architecture enters the room.

Rung one: replication. Copy the data onto more machines.

• Leader–follower — all writes go to one node; reads spread across the copies. Fits the web, where almost everything is read far more than written. The catch is replication lag: a user who writes and immediately reads may not see their own change — that is where "I posted a comment and it disappeared" tickets come from. The usual cure is read-your-writes routing: serve a user's reads from the leader briefly after they write.
• Leader–leader — both sides take writes; survives losing a whole region. But the same row can now be edited in two places at once, and that is a conflict you must resolve — the lazy answer, "last write wins," silently throws data away. Earns its keep across regions; inside one region it is usually complexity with no customer.

Rung two: split the data.

• Federation (vertical split) — users, photos, and payments in separate databases. Coarse, simple, delays the harder move.
• Sharding (horizontal split) — one table's rows spread across machines:

Each shard is a complete database for a slice of the users. This is the move that finally scales writes, and it charges honestly for it:

• Queries that cross shards now live in your application code.
• One celebrity account can overload a single shard (the hot key problem).
• With naive hash mod n routing, changing the number of shards moves nearly every key — which is why consistent hashing exists: it arranges keys so adding or removing a node only moves the keys next to it.

Instagram's version is worth knowing: thousands of logical shards in Postgres mapped onto a few physical machines — "resharding" means moving logical shards between machines, and the count n never changes. Plain Postgres underneath; all the cleverness lives in the key.

Sharding also breaks the humblest tool you had: auto-increment IDs, which only count correctly on one machine. The standard replacement is a Snowflake-style ID — timestamp, then machine or shard number, then a per-machine sequence, packed into 64 bits. No coordinator to call, roughly sortable by creation time, and unique across the fleet. Instagram's IDs are this exact shape with the logical shard baked into the middle.

And the tempting shortcut: denormalization — copying data around to avoid joins. Often right at scale (it is what makes sharding livable), but every copied fact must now be kept true in several places at once. You trade cheap reads for a more complicated write path — and once your queries cannot reach two tables on one machine anyway, the trade stops being optional.

SQL or NoSQL

The honest default is boring: start with Postgres. A relational database gives you transactions, joins, constraints, and decades of tooling. At small-to-startup scale it is almost never the bottleneck — and every guarantee it gives you for free becomes your own application code the moment you leave.

The vocabulary behind the split:

ACID — what a relational transaction promises: atomic, consistent, isolated, durable. All or nothing, even mid-crash. BASE — the looser deal most NoSQL settles for: basically available, soft state, eventually consistent.

NoSQL is not one thing. It is four different bargains:

Notice the pattern in every row: NoSQL asks you to know your access patterns in advance, and to shape the storage around them, in exchange for scale on exactly those paths. Great trade for one known, hot, simple path; bad trade for product queries that keep evolving. This is why real architectures mix stores — Discord keeps trillions of chat messages in ScyllaDB (wide-column, write-heavy, one access pattern: recent messages for a channel) while relational data lives elsewhere. Search is the same story: text lookup wants an inverted index — a map from each word to the documents containing it — which is why products bolt on Elasticsearch, fed from the primary database, instead of running LIKE queries at scale. The interview answer is never "NoSQL because scale." It is naming the access pattern that justifies the bargain.

One last vocabulary pair for the data layer:

OLTP — many small, fast reads and writes; the store serving the product. OLAP — few huge queries over history; the warehouse answering questions about the product.

Keep them on separate systems. One analyst's year-long scan should never sit in front of a customer's checkout.

CAP, and what it really says

CAP is usually quoted as "pick two of three," which gets it wrong. In a distributed system you do not get to opt out of network partitions — the network will split sometimes, like weather. The real statement is narrower and more useful: when a partition happens, you choose what to give up.

• CP — refuse to answer rather than risk answering wrong. ZooKeeper and etcd behave this way, which is exactly what you want from the thing that decides who holds a lock.
• AP — answer from whichever side of the split you are on, clean up later. Amazon's Dynamo (and Cassandra after it) chose this on purpose: for a shopping cart, a slightly stale cart beats an error page, and conflicting writes can be merged after the network heals.

PACELC is the follow-up worth knowing because it covers the other 99.9% of the time: else — when there is no partition — you still trade latency against consistency on every replicated write. Waiting for all replicas to confirm is consistency paid in milliseconds; confirming early is speed paid in staleness.

And consistency is a spectrum, not a switch:

• Weak — after a write, reads may or may not see it, no promise at all. Acceptable where the moment matters more than the record: live video, voice calls, game state.
• Eventual — replicas converge if writes stop. Fine for likes, view counts, presence dots.
• Read-your-writes / monotonic reads — the session guarantees users actually notice: you always see your own changes, and time never runs backwards between two refreshes.
• Strong — the system behaves like one machine. Costs coordination; banks pay it, and Google built Spanner with GPS and atomic clocks in its datacenters because they wanted it at global scale badly enough to buy hardware for it.

Most products need strong consistency for a few operations and eventual for the rest. The design skill is saying which is which, per operation, out loud.

Caching

Cache — a small, fast store holding copies of expensive results so repeat reads skip the expensive part.

Caching is how systems survive their own success, and it is layers, not a layer: the browser's cache, the CDN, an application cache like Redis, the database's own memory. At every layer the question is the same: how much cheaper does this make the read, and how much staleness can the product accept in return?

The strategies, in the order you meet them:

• Cache-aside — app checks the cache, misses, reads the database, fills the cache. The workhorse: only data actually read gets cached, and a cold cache means slow, not down.
• Write-through — every write updates cache and database together. Reads are never stale; every write pays latency for data that may never be read.
• Write-behind — confirm the write immediately, flush to the database later. Fastest, with a sharp edge: crash before the flush and you lose writes you already confirmed.
• Refresh-ahead — the cache renews hot keys just before they expire. Smooths latency spikes for predictably hot data; wasted work when the prediction is wrong.

Eviction is its own decision, and LRU — drop the least recently used — is the default for a good reason: recency is the best cheap predictor of the next read. TTLs put a ceiling on staleness; adding jitter stops a thousand keys expiring in the same second.

Then the two famous hard problems:

• Invalidation — the cache and the database disagree, and every strategy above is just a different choice of when you find out.
• Thundering herd — a hot key expires and ten thousand requests all miss at once, all rebuilding it together and flattening the database the cache was protecting. Facebook's memcache fleet handles this with leases: the first miss gets a token and the job of rebuilding; everyone else briefly gets the stale value. The general form — let one worker rebuild, make everyone else wait or accept stale.

Experienced teams also cache negative results ("no such user") — otherwise anyone asking for missing keys walks straight past the cache to the database.

Asynchronous work

Message queue — a buffer that holds work so the sender can move on before the receiver has done it.

The rule: a request should only do the work the user is actually waiting for. Everything else — transcoding the video, resizing the photo, sending the email, updating recommendations — belongs behind a queue.

The API accepts the upload, drops a job on the queue, and returns 202 Accepted in milliseconds. Workers pull jobs at their own pace. The queue's quiet superpower is separating the two sides in time: a traffic spike becomes a longer queue instead of a wave of timeouts, and deploying a new worker version is invisible to users.

Two facts separate people who have run queues from people who have read about them:

• Delivery is at-least-once in every practical system — true exactly-once delivery is a whiteboard myth. A message will occasionally arrive twice, so handlers must be idempotent: doing the work twice gives the same result as doing it once. Stripe makes the pattern visible from the outside — clients send an idempotency key, so retrying a payment that timed out cannot charge the card twice.
• An unbounded queue is not resilience. A queue that only ever grows is delaying your outage, not preventing it — that growth is back pressure made visible, and the honest answers are more workers, dropping low-value work, or slowing the producer.

One vocabulary split worth having:

• Message queues (RabbitMQ, SQS) — hand each job to one worker, then delete it. Task distribution.
• Streams (Kafka) — keep an ordered log that many consumers read at their own positions. The same events can feed analytics, search indexing, and caches at once, which is why Kafka became the connective tissue of so many data platforms.

Talking between services

Underneath everything sit two transports:

• TCP — reliable, ordered delivery, paid for with handshakes and waiting on lost packets. The default for almost everything.
• UDP — send and hope. Not the cheap option but the correct one when a late packet is worthless: live video, game state, DNS lookups. A video frame that arrives late is worse than one that never arrives.

At the API level:

• REST over HTTP/JSON — the common language: readable, cacheable, supported everywhere. The right default at any boundary people or third parties touch.
• gRPC — typed contracts and a compact binary format, at the cost of that universality. The right trade on internal hot paths where you own both ends.
• GraphQL — solves one specific problem (many kinds of client, each wanting different fields from many resources) and brings its own caching and query-cost headaches. A scalpel, not a default.

However services talk, the failure rules are the same, and this short list prevents most cascading outages:

• Every call has a timeout.
• Retries are limited, spaced out exponentially, with jitter — a thousand clients retrying on the same schedule is an attack you run on yourself.
• Anything retried is idempotent.
• A dependency that keeps failing gets a circuit breaker: stop calling it, serve something reduced, check back until it heals. Partial degradation beats total collapse — but only if you designed the reduced mode before you needed it.

Real-time delivery

Interviews love chat apps and live dashboards, so know the four ways a server gets news to a browser:

• Polling — the client asks every few seconds. Simple, mostly wasted requests, latency up to the polling gap.
• Long polling — the client asks and the server holds the request open until there is news. Less waste, still one request per message.
• Server-sent events (SSE) — one long-lived HTTP response the server streams events down. One direction only; plain HTTP, so it passes through proxies well. Right for feeds and dashboards.
• WebSocket — one persistent two-way connection. Right for chat and games; the cost is that every open socket is state a server holds, which makes the tier harder to scale and drain.

The rough rule: WebSocket when the client must send often too; SSE when the client mostly listens; polling when the update rate is slow and simplicity wins.

Security, the minimum

Security is a whole field, but the floor is short:

• TLS on every connection, external and internal.
• Passwords hashed with a slow, salted algorithm — bcrypt or argon2, never a general-purpose hash.
• Validate input at the boundary; parameterized queries, always.
• Least privilege: every service gets only the access it needs, so one stolen credential does not hand over the fleet.
• Rate-limit anything a stranger can call.
• The rule that contains the others: do not build your own crypto, and do not build your own auth — the industry has deep scars from both.

Observability

You cannot operate what you cannot see, and interviewers increasingly ask how you would know your design is healthy. Three signals, three different questions:

• Metrics — numbers over time (request rate, error rate, latency percentiles). Cheap to keep, first to alert on.
• Logs — what happened, one event at a time. Expensive at volume; invaluable in the incident.
• Traces — one request's journey across every service it touched. The only way to find which hop in a microservice chain ate the latency budget.

The four golden signals — latency, traffic, errors, saturation — are a complete first dashboard for almost any service, and "alert on p99 and error rate, not averages" is the one-line version of a mature monitoring philosophy.

In the interview room

A system design interview is a forty-five-minute simulation of the first week of a project, and what is really being graded is not whether you drew the right boxes — it is whether you make trade-offs out loud. A shape that works:

1. Scope it (about five minutes). Which features are in and out? How many users, how many requests per second, how much data? Reads or writes heavier? How fresh must data be? Half the value here is showing you do not design before you understand.
2. Run the estimate. The thirty seconds of arithmetic from the top of this article. It decides read-heavy vs write-heavy, cache-first vs database-first, one region or several.
3. Draw the skeleton. The canonical shape, adapted to this problem. Say what each box is for in this design — a box you cannot justify is a box you should erase.
4. Go deep where it hurts. Every good question has one hard spot — the fan-out in a feed, deduplication in a crawler, hot keys in a counter, ordering in chat. Find it, name it, and spend your remaining time there. Naming the bottleneck before the interviewer does is the difference between driving and being driven.

And through all of it, say the trade-off at every fork. "Cache-aside, because a cold cache makes us slow instead of wrong, and this product can tolerate staleness" beats a silently drawn Redis box in any interview on earth.

The classic practice questions are each worth doing for the specific muscle they isolate:

• URL shortener — key generation, huge read volume.
• News feed — fan-out on write vs fan-out on read; Twitter's hybrid, where celebrity posts fan out on read instead, is the canonical deep-dive.
• Chat system — ordering, presence, delivery guarantees, and the real-time options above.
• Web crawler — politeness, deduplication, managing the frontier. Deduplication at billions of URLs is where the Bloom filter earns its fame: a tiny probabilistic set that answers "definitely new" or "probably seen," never storing the URLs themselves.
• Rate limiter — token bucket, and where the counters live.
• Nearby search (drivers, restaurants) — geospatial indexing: geohashes or quadtrees turn "within 2 km" into a key lookup.
• Video platform — chunked uploads to object storage, transcoding behind the queue, delivery through the CDN. Three sections of this article meeting in one product.

Where to go deeper

This article is a map, and maps are for leaving. The system design primer is the base this piece grew from, and still the best single collection of the fundamentals. Designing Data-Intensive Applications (Martin Kleppmann) is the book that turns the storage and consistency sections above from vocabulary into understanding — the best technical book I know, full stop. Alex Xu's System Design Interview volumes are the closest thing to rehearsal. And the primary sources are more readable than their reputations suggest: the Dynamo paper for AP thinking, Facebook's memcache paper for caching at scale, and the engineering blogs at Netflix, Discord, Stripe, and Instagram for what these trade-offs look like with money on the line.

The numbers first, the skeleton second, the trade-offs always.