Scalable Systems

Systems design is the art of arranging servers, databases, caches, and networks into an architecture that stays fast, correct, and available as it grows from a hundred users to a hundred million. Where the earlier parts of this book asked "how does one machine work?", this part asks "how do many machines work together?" — and the answer is a small vocabulary of building blocks that recur in every large system.

Every design is judged against four properties, and they constantly trade off against one another:

• Scalable — handles growth in users and data without a rewrite.
• Reliable — keeps working correctly even when components fail.
• Available — responds to requests, even under load or partial failure.
• Maintainable — easy to understand, change, and operate.

Scalability — Up vs Out

There are only two ways to handle more load: get a bigger machine, or get more machines.

Vertical scaling (a bigger box) is simple — no code changes — but it hits a hard ceiling and leaves you with a single point of failure. Horizontal scaling (more boxes) is effectively unlimited and more resilient, but it forces you to confront the central challenge of this entire part: coordinating state across machines. Almost everything that follows exists to make horizontal scaling work.

Load Balancing

The instant you have more than one server, something must decide which one handles each request. That's a load balancer — it spreads incoming traffic across a pool of identical servers, and reroutes around any that fail.

It chooses a server using one of a few algorithms:

Algorithm	How it works
Round robin	Cycle through servers in order
Least connections	Send to the server with the fewest active connections
IP hash	Hash the client IP → same client always hits the same server (session stickiness)
Weighted	Bigger servers receive proportionally more traffic

Analogy: A load balancer is the host at a restaurant, seating arriving guests across tables so no single waiter is overwhelmed — and skipping any table that's out of service.

Caching

A cache stores frequently used data somewhere faster than the source — usually memory instead of disk or network. It's the highest-leverage performance tool in systems design.

Caching can happen at every layer of a stack:

• Client-side — browser and mobile app caches
• CDN — static content served from near the user (below)
• Application — Redis or Memcached holding hot objects
• Database — query cache and buffer pool

The hard part isn't storing data — it's knowing when the cached copy is stale. The strategy you pick is a trade between speed and freshness:

Strategy	How it works	Trade-off
Write-through	Write to cache and DB together	Slower writes, always consistent
Write-behind	Write to cache, flush to DB async	Fast writes, risk of loss on crash
Cache-aside	App reads cache, falls back to DB on miss	Most common, risk of stale reads
TTL	Entries expire after N seconds	Simple, eventually consistent

Phil Karlton: "There are only two hard things in Computer Science: cache invalidation and naming things."

Analogy: A cache is a cheat sheet of formulas you keep handy during an exam. You glance at it instead of re-deriving each time — but if a formula changes and you don't update the sheet, you'll confidently write the wrong answer (a stale cache).

CDN — Content Delivery Network

A CDN is caching applied to geography. It's a fleet of servers spread across the globe that cache static content (images, CSS, JS, video) and serve it from an edge location physically close to each user. Since latency is bounded by the speed of light, distance is destiny:

Analogy: A CDN is a restaurant franchise. Instead of every customer flying to the one original location in New York, there's a branch in every city serving the same menu.

Proxies

A proxy is an intermediary that requests pass through — and which way it faces changes everything.

Analogy: A forward proxy is a lawyer speaking on your behalf, hiding your identity from the other side. A reverse proxy is a company receptionist — clients talk to reception, never to individual employees, who stay hidden behind it.

A load balancer is one kind of reverse proxy; so is the TLS-terminating, caching front door (Nginx, Cloudflare) that sits before nearly every production web service.

Message Queues — Asynchronous Communication

When one service calls another directly and waits for the reply, the two are tightly coupled: if the callee is slow or down, the caller is stuck. A message queue breaks that coupling by letting services communicate through an intermediary.

Decoupling through a queue buys you four things: services don't need to know about each other, the queue buffers traffic spikes, messages persist if the consumer is temporarily down, and you can scale throughput by adding more consumers.

Analogy: A queue is a mailbox. The sender drops a letter and walks away; the carrier picks it up whenever ready. Neither has to be present at the same moment.

Kafka deserves a special mention because it's more than a queue — it's a distributed, append-only commit log. Messages are ordered, durably stored, and replayable, and different consumers can read from different positions independently.

Analogy: A plain queue is a bakery line — once you're served, you're gone. Kafka is a newspaper archive — new issues are appended, old issues stay readable, and every reader can be at a different point in the archive.

Beyond the big architectural patterns, a handful of specialized building blocks show up again and again in real systems — the components that protect, accelerate, and scale them. Each is a small, sharp tool worth knowing by name, because reaching for the right one is often the entire solution to a design problem.

Rate Limiting

Any public-facing service needs to protect itself from abuse, runaway clients, and traffic spikes by capping how many requests a client may make. The most popular algorithm is the token bucket:

It's favored because it permits short bursts (drain the full bucket at once) while still bounding the sustained rate (the refill speed). The alternatives trade flexibility for simplicity:

Algorithm	How it works
Token bucket	Tokens refill at a fixed rate; each request spends one; empty = rejected
Sliding window	Count requests in the trailing N seconds
Leaky bucket	Requests queue and drain at a fixed rate, smoothing bursts away

Analogy: The token bucket is a dripping faucet filling a bucket. Each request scoops out some water; a full bucket lets you scoop a lot at once (a burst), but you can never use water faster than it drips in over the long run.

Bloom Filters

Sometimes you only need to answer "is this definitely not in the set?" — and you need to answer it using almost no memory. A Bloom filter is a probabilistic structure that does exactly that:

It can say "definitely not present" with certainty, or "probably present" with a small false-positive chance — and it never produces a false negative. The payoff is enormous space savings.

Analogy: A Bloom filter is a doorman who occasionally waves through someone not on the list (a false positive) but never turns away someone who genuinely is on it (no false negatives).

It's used everywhere a cheap pre-check can avoid an expensive lookup: databases skip disk reads for keys that can't exist, CDNs avoid origin fetches, and browsers check URLs against malware lists.

Consistent Hashing with Virtual Nodes

Basic consistent hashing (Chapter 23) has a weakness: with only a few servers, or servers of differing capacity, the ring divides unevenly and some servers get far more load than others. The fix is virtual nodes — represent each physical server by many points scattered around the ring:

Physical:  Server A,  Server B
On ring:   A·1  B·1  A·2  B·2  A·3  B·3   (interleaved, evenly spread)

With each server owning many small arcs instead of one big one, load evens out, and adding or removing a server reshuffles a smooth fraction of keys rather than a lumpy one.

LSM Trees — Built for Heavy Writes

The B-tree indexes of Chapter 19 are optimized for reads and update data in place — costly when writes dominate. Write-heavy databases (Cassandra, LevelDB, RocksDB) instead use Log-Structured Merge trees, which turn random writes into fast sequential ones:

Analogy: An LSM tree is jotting notes on sticky pads (instant writes to memory), then periodically filing them into sorted folders (compaction to disk). Writing is effortless; the organizing happens later, in bulk.

The trade-off is read amplification — a lookup may have to check several files — which is exactly why LSM trees lean on bloom filters to skip files that can't contain the key. It's a perfect illustration of how these building blocks compose: the right system is usually several of them working together.

Once a system and the team building it grow large enough, a single codebase can become a bottleneck — every change requires coordinating with everyone, and the whole thing deploys (and breaks) as one. Microservices are the architectural response: split the system into small, independently deployable services. But they trade one set of problems for another, and the patterns in this chapter exist to manage that trade.

Monolith vs Microservices

The single most important rule in this entire chapter: start with a monolith. A monolith is simpler to build, test, deploy, and reason about. Split into microservices only when you have a concrete reason — independent teams that need to deploy on their own schedules, components with wildly different scaling needs, or a codebase that's genuinely too large to hold in one head. Reaching for microservices on day one buys you distributed-systems complexity (everything in the previous chapter) before you have the scale to need it.

API Gateway

With many services behind it, you don't want clients talking to each of them directly. An API gateway is a single front door that routes each request to the right service:

Centralizing here lets you handle cross-cutting concerns once — authentication, rate limiting, request aggregation, protocol translation — instead of reimplementing them in every service.

Analogy: The API gateway is a hotel concierge. Guests make every request through the concierge, who routes them to the right department — they never wander into the kitchen or the laundry themselves.

Service Discovery

Services start, stop, and move constantly — instances come and go with autoscaling and deploys. So how does Service A find a healthy instance of Service B? Through a service registry — a live directory that each instance registers with on startup:

auth-service  →  [10.0.1.1:8080, 10.0.1.2:8080]
order-service →  [10.0.2.1:9090]

In client-side discovery the caller queries the registry (Consul, etcd, ZooKeeper) and picks an instance itself; in server-side discovery a load balancer does it on the caller's behalf. Either way, hardcoded addresses are replaced by a lookup against the current reality.

Circuit Breaker

When Service B starts failing, the worst thing Service A can do is keep hammering it with requests — that wastes resources, piles up timeouts, and can cascade into a system-wide outage. The circuit breaker pattern borrows from electrical engineering:

When failures cross a threshold the breaker trips open and requests fail instantly (no waiting on a doomed call). After a cooldown it goes half-open, letting a few test requests through; if they succeed it closes and normal traffic resumes, otherwise it trips open again.

Analogy: It's the breaker in your home's electrical panel. On an overload it trips to prevent a fire; after a while you reset it; if the fault is gone, power flows normally again.

Saga — Transactions Across Services

A single database gives you ACID transactions for free. Across microservices, each owning its own database, there's no shared transaction to roll back. A saga is the answer: a sequence of local transactions, each paired with a compensating action that undoes it.

There are two ways to coordinate a saga. In orchestration, a central coordinator explicitly tells each service what to do and when. In choreography, there's no conductor — each service listens for events and reacts, triggering the next step.

Analogy: Choreography is dancers who each know their part and respond to one another. Orchestration is a conductor cueing each musician. Choreography is more decoupled; orchestration is easier to follow and debug.

We scale horizontally because we have no choice — single machines have ceilings, they fail, and our users are spread across the planet. But the moment state lives on more than one machine connected by an unreliable network, a set of genuinely hard problems appears that have no equivalent on a single computer. This chapter is about those problems and the ideas that tame them.

The Fallacies of Distributed Computing

Peter Deutsch's famous list names the assumptions engineers unconsciously carry over from single-machine programming — every one of them false in a distributed system:

1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

Internalizing that all eight are lies is the mental shift from writing application code to designing distributed systems. Every pattern below is, at bottom, a defense against one of these realities.

Consistency Models

If data is replicated across machines, how soon after a write do all readers see it? The answer is a choice, expressed as a consistency model:

Between the two extremes lie useful middle grounds:

Model	Guarantee	Example
Strong (linearizable)	Reads always return the latest write	ZooKeeper, single-node RDBMS
Sequential	All nodes see operations in the same order	—
Causal	Cause-and-effect operations stay ordered	collaborative editors
Eventual	All replicas converge, given time	DynamoDB, DNS, Cassandra

Analogy: Strong consistency is a shared whiteboard — write on it and everyone sees the change instantly. Eventual consistency is filing a change of address — old mail keeps arriving at the old place for a while, but everything redirects eventually.

Consensus — Agreeing When Nothing Is Reliable

How do several nodes agree on a single value when messages can be delayed, lost, or duplicated and nodes can crash? This is the consensus problem, and it underpins leader election, distributed locks, and replicated databases. The dominant practical algorithm is Raft:

The key idea is majority quorum: a decision is final once more than half the nodes agree. With 3 nodes the majority is 2, so the cluster tolerates 1 failure; with 5 nodes it tolerates 2. Paxos is the older, mathematically rigorous algorithm that solves the same problem — equally correct, famously harder to understand, which is precisely why Raft was designed.

Analogy: Raft is a classroom electing a president. All decisions go through that one student; if they're absent, the class holds a new election; and a decision is only official once a majority of hands go up.

Time and Ordering

On one computer, "what happened first?" is trivial. In a distributed system there is no global clock — every machine's clock drifts slightly — so ordering events is shockingly hard. Three approaches:

Lamport timestamps are a logical clock: a simple counter that guarantees if event A caused event B, then timestamp(A) < timestamp(B). It captures causality without any reference to real time.

Vector clocks go further — each node keeps a counter for every node, which lets the system detect events that are genuinely concurrent (neither caused the other):

Node A: [2, 0, 0]   "I've done 2 things, heard nothing from B or C"
Node B: [1, 3, 0]   "I saw A's 1st event; I've done 3 of my own"
Node C: [1, 2, 1]   "I saw A's 1st and B's 2nd; I've done 1"

Google's TrueTime (used by Spanner) takes a different tack entirely: put atomic clocks and GPS receivers in every data center to keep clocks within a known uncertainty bound (~7 ms), then wait out that uncertainty. It buys global strong consistency by throwing hardware at a software problem.

Analogy: Three people write a story in separate rooms, occasionally passing notes. Lamport timestamps number each note. Vector clocks have each person tally how many notes they've seen from everyone — enough to notice when two of them edited the same scene without knowing about each other.

Consistent Hashing

When you shard data across servers, the naive scheme server = hash(key) % N has a fatal flaw: change N (add or remove one server) and almost every key moves to a new server — a catastrophic reshuffle. Consistent hashing fixes this by placing both keys and servers on a ring:

Analogy: Divide a clock face among three people, each responsible for the minutes nearest their position. Adding a fourth person only steals work from their immediate neighbors — the rest of the clock is untouched.

This is the backbone of distributed caches and databases like Cassandra and DynamoDB — and Chapter 25 adds the refinement (virtual nodes) that keeps the load evenly balanced.

Chapter 23 introduced consensus with Raft. Here we go deeper — into why agreement is fundamentally hard, what happens when participants actively lie, and the coordination services that production systems lean on so they don't have to implement any of this themselves.

The Two Generals Problem

Start with the impossibility result that frames everything. Two generals must agree to attack at the same time, but can only communicate by messengers who might be captured. Can they ever be certain they agree?

The answer is no. Any message confirming agreement itself needs confirmation, and that confirmation needs confirming — an infinite regress. Perfect agreement over an unreliable channel is impossible. This is why real protocols (like TCP's handshake) settle for practically reliable — good enough with retransmissions — rather than perfect.

The Byzantine Generals Problem

Two Generals assumes honest participants who might simply fail to communicate. What if some are traitors, actively sending conflicting messages? A loyal general hears "attack" from one peer and "retreat" from the same peer relayed through another. Can the loyal generals still agree?

Remarkably, yes — but only with enough of them. The result is that with 3f + 1 participants you can tolerate f traitors (so 4 generals survive 1 traitor). Protocols that achieve this are called Byzantine Fault Tolerant (e.g. PBFT), and they underpin some blockchains and safety-critical infrastructure.

Analogy: A crash fault is a worker who might not show up. A Byzantine fault is a worker who might actively sabotage the project. Defending against the second is far harder and more expensive — which is why most systems assume only crash faults, making Raft and Paxos sufficient.

ZooKeeper and etcd — Coordination as a Service

Most teams shouldn't implement consensus themselves. Instead they delegate it to a hardened coordination service that exposes a simple, file-system-like interface over a consensus core. ZooKeeper and etcd provide the primitives distributed systems need most:

• Configuration that every node can read consistently
• Leader election — who's in charge right now?
• Distributed locks — only one process touches a resource at a time
• Service discovery — where are the live instances of service Y?

Their data model is a hierarchy of small nodes, like a tiny filesystem:

/config
    database_url   = "postgres://..."
    feature_flags  = "{...}"
/leaders
    order-service  = "node-3"
/locks
    inventory-update = "node-7"   (ephemeral)

Two features make this powerful. Ephemeral nodes vanish automatically when the client that created them disconnects — perfect for leader election, since a dead leader's node disappears and triggers a new one. Watches let clients subscribe to a node and be notified the instant it changes.

Analogy: ZooKeeper is a trusted notary that all parties agree to consult. Who's in charge? Ask the notary. Need exclusive access? Get a lock from the notary. Need shared config? Store it with the notary.

etcd is the modern equivalent that backs Kubernetes — a flatter key-value API built on Raft, solving the same coordination problems with a simpler interface.

As applications grow data-hungry, a handful of patterns recur for keeping data correct, fast, and flowing between systems. They share a common thread: treating change itself — the stream of events — as a first-class citizen, rather than just the current state.

Event Sourcing

The traditional approach stores only current state: Alice's balance is $150. But how did it reach $150? That information is gone. Event sourcing instead stores the immutable sequence of events, and derives current state by replaying them:

The benefits are substantial: a complete audit trail, the ability to reconstruct any past state, and freedom to build many different "views" from the same events. The costs are real too — event schemas must evolve carefully, and replaying a long history is slow (solved with periodic snapshots).

Analogy: Traditional storage is a bank statement showing only today's balance. Event sourcing is the full transaction history — you can always derive the balance from the transactions, but never the reverse.

CQRS

Command Query Responsibility Segregation splits the write path from the read path, because the two have opposing needs:

Writes need validation, business rules, and consistency; reads need speed, flexible shapes, and denormalization. CQRS pairs naturally with event sourcing — commands append events, which are then projected into read-optimized views.

Analogy: In a newspaper, the editorial team (writes) carefully composes and reviews articles, while the printing press (reads) churns out thousands of copies in a format optimized purely for reading. Entirely different concerns, deliberately separated.

Change Data Capture

How do you keep a search index, a cache, and a data warehouse all in sync with your primary database — without every system hammering it with polling queries? Change Data Capture taps the database's own change log and streams every insert, update, and delete to whoever needs it:

Analogy: CDC is a security camera on your database. Every change is recorded and broadcast to whoever needs to know — so instead of each downstream system repeatedly asking "anything new?", the changes flow to them automatically.

There are two fundamentally different ways to process data at scale, distinguished by when the work happens relative to the data arriving.

Batch Processing — MapReduce and Spark

Batch jobs run over a large, bounded dataset all at once. The model that made this scalable was Google's MapReduce (2004): split the work into a map step that runs in parallel across machines, a shuffle that groups results by key, and a reduce step that aggregates each group. The canonical example is counting every word on the web:

"the cat sat on the mat"
  → map:     (the,1) (cat,1) (sat,1) (on,1) (the,1) (mat,1)
  → shuffle: the→[1,1]  cat→[1]  sat→[1] ...
  → reduce:  the→2  cat→1  sat→1

Apache Spark is MapReduce's successor: the same idea, but it keeps intermediate data in memory between steps instead of writing to disk, making it 10–100× faster for iterative workloads.

Analogy: MapReduce is a census. Workers fan out to every neighborhood (map), bring data back, you group it by category (shuffle), and tally the totals (reduce).

Stream Processing — Flink and Kafka Streams

Streaming flips the model: instead of waiting for data to accumulate, you process each event the instant it arrives, maintaining running state (counts, fraud scores, dashboards) in memory with periodic checkpoints to disk. The hard parts are all about time:

• Event time vs processing time — when something happened vs when your system saw it (they differ when events arrive late or out of order).

• Watermarks — a heuristic declaring "I believe all events before time T have now arrived," so a window can close.

• Windowing — how you group an unbounded stream into finite chunks to aggregate:

• Exactly-once processing — the difficult guarantee that each event affects the result precisely once, despite retries and failures.

Analogy: Batch is grading every exam at the end of the semester; streaming is grading each assignment as it's handed in; windowing is grading by week.

Lambda vs Kappa

How do you get both the accuracy of batch and the freshness of streaming? Two architectures answer differently:

Lambda runs both pipelines and merges them, accepting the burden of two codebases. Kappa bets that a single, replayable stream is enough — if you need to recompute, you just replay the log from the beginning. The rise of durable logs like Kafka has made Kappa increasingly practical.

Searching billions of documents for a phrase in milliseconds sounds impossible — until you see the data structure that makes it routine. Search engines like Elasticsearch (built on Lucene) are, at their core, one big inverted index plus a ranking function.

The Inverted Index

A forward index maps each document to the words it contains — the obvious layout, and useless for search (you'd scan every document). Flip it around and you get the inverted index: each word maps to the documents that contain it.

Now a query is a set intersection over short lists, not a scan over millions of documents.

Analogy: A forward index is reading every book to find which mention "dinosaurs." An inverted index is the index at the back of a book — look up "dinosaurs," get the page numbers instantly.

Ranking with TF-IDF

Matching isn't enough; results must be ranked. The classic measure is TF-IDF, the product of two intuitions:

• Term Frequency (TF) — how often the term appears in this document. More occurrences suggest more relevance.
• Inverse Document Frequency (IDF) — how rare the term is across all documents. "the" appears everywhere (low IDF, not distinctive); "quantum" is rare (high IDF, very distinctive).

A high TF × IDF score means the term is frequent here yet rare overall — a strong signal this document is about that term. Modern engines extend this with BM25 and, increasingly, semantic vector similarity, but the intuition is the same.

The Analysis Pipeline

Raw text isn't indexed directly — it first passes through an analysis pipeline that normalizes it so that "Running," "runs," and "ran" all match a search for "run":

The same analysis is applied to both documents (at index time) and queries (at search time), so the two always line up.

Scaling Out

A single index can't hold the whole web, so it's split into shards (each an independent Lucene index) spread across nodes, with replicas for availability:

Analogy: Searching a library of millions of books, you don't keep one giant card catalog. You split it into sections (shards), each managed by a librarian (node); a query goes to all of them at once, and their answers are merged and ranked.

Threads with shared memory and locks — the model most programmers learn first — are also the hardest to get right. Shared mutable state invites race conditions, deadlocks, and bugs that appear only under load and vanish when you attach a debugger. So languages and runtimes have evolved several other models for doing many things at once, each attacking the problem from a different angle.

Concurrency vs parallelism. They're often confused. Concurrency is structuring a program as independent tasks that can make progress in overlapping time periods — a property of the design. Parallelism is actually running things simultaneously on multiple cores — a property of the hardware. A single-core machine can be highly concurrent (an event loop juggling thousands of connections) without any parallelism. The models below are about concurrency; whether they run in parallel is a separate question.

The Event Loop (Node.js, Nginx)

A single thread that never blocks. Every I/O operation is asynchronous: instead of waiting for a disk read or network reply, the thread registers a callback and moves on to the next ready task. When the I/O completes, its callback is queued and eventually run.

Analogy: A single waiter who never stands idle. They take table 1's order, hand it to the kitchen, and immediately move to table 2 — then table 3 — and deliver food whenever the kitchen rings the bell (I/O complete). One waiter serves many tables. But if they spend ten minutes chatting with one table (CPU-intensive work), everyone else waits — the event loop's fatal weakness.

The Actor Model (Erlang, Akka)

Eliminate shared state entirely. Each actor is an independent unit that owns its private state and communicates only by sending messages.

Because nothing is shared, there are no locks and no data races — by design. This also extends naturally across machines: sending a message to an actor on another server looks the same as sending one locally. Erlang pairs this with a famous "let it crash" philosophy: rather than defensively guarding against every error, you let a failing actor die and have a supervisor restart it into a known-good state. This is how telecom systems reach nine 9's of uptime (~31 ms of downtime per year).

Analogy: A company where every employee works in a private office (own state) and communicates only through memos in their inbox (mailbox). Nobody barges in to rummage through someone else's desk — so two people can never corrupt the same data at once.

CSP — Communicating Sequential Processes (Go)

Go shares the actors' creed — don't communicate by sharing memory; share memory by communicating — but the unit of communication is the channel, a typed pipe, rather than a named actor. Lightweight goroutines (thousands are cheap) send and receive on channels:

func producer(ch chan<- int) {
    for i := 0; i < 10; i++ {
        ch <- i        // send to channel
    }
    close(ch)
}

func consumer(ch <-chan int) {
    for val := range ch {   // receive until channel closes
        fmt.Println(val)
    }
}

func main() {
    ch := make(chan int, 5) // buffered channel
    go producer(ch)         // launch a goroutine
    consumer(ch)
}

Analogy: Channels are the pneumatic tubes in an old bank. Tellers (goroutines) send capsules (messages) through tubes (channels), and each tube is typed — the cash tube carries only cash. The tube itself handles all the synchronization.

Async/Await (Python, JavaScript, Rust, C#)

Not a new concurrency engine so much as syntax that makes asynchronous code readable. It turns a nest of callbacks into something that reads top-to-bottom:

# Callback hell — nested and hard to follow:
fetch_user(id, lambda user:
    fetch_orders(user.id, lambda orders:
        fetch_items(orders[0].id, lambda items:
            print(items))))

# Async/await — flat and linear:
async def get_items():
    user   = await fetch_user(id)
    orders = await fetch_orders(user.id)
    items  = await fetch_items(orders[0].id)
    print(items)

Each await yields control back to the event loop, which runs other tasks while this one waits for I/O — so the code reads as if it blocks, but the thread stays busy.

Analogy: await is putting a bookmark in your book when the doorbell rings. You answer the door (another task) and return to exactly where you left off. It reads like you waited, but you were productive the whole time.

Choosing a Model

Model	Shared state?	Communication	Strengths	Used by
Threads + locks	Yes (dangerous)	Shared memory	Familiar, fine-grained control	C, C++, Java
Event loop	No (single thread)	Callbacks/events	High I/O concurrency, simple	Node.js, Nginx
Actor model	No (isolated)	Messages	Fault tolerance, distribution	Erlang, Akka
CSP	No (mostly)	Channels	Structured concurrency	Go
Async/await	Varies	Futures/promises	Readable async code	Python, JS, Rust, C#

The thread running through all of them: the more you avoid shared mutable state, the fewer ways your program can corrupt itself.

For decades, programs got faster simply because CPU clock speeds rose. That free lunch ended around 2005 — physics (heat and power) capped clock speeds — and the industry pivoted from making one core faster to packing in more cores. Today the most dramatic performance gains come from a fundamentally different kind of chip: the GPU.

CPU vs GPU

The difference is a deliberate trade. A CPU spends its transistor budget on a few powerful cores with deep caches and sophisticated branch prediction — optimized for latency, finishing any single complex task quickly. A GPU spends the same budget on thousands of simple cores — optimized for throughput, doing the same operation across enormous amounts of data at once.

Analogy: A CPU is a few brilliant professors who can solve any problem. A GPU is an army of students who each handle simple arithmetic. To grade 10,000 multiple-choice tests, the army wins overwhelmingly; to write one research paper, you need a professor.

SIMD — One Instruction, Many Data

Even a single CPU core exploits data parallelism through SIMD (Single Instruction, Multiple Data): one instruction applied to a whole vector of values at once.

A GPU is, in essence, this idea taken to an extreme and replicated across thousands of cores.

Why GPUs Power Machine Learning

Training a neural network is, at its core, an avalanche of matrix multiplication — the same multiply-and-add applied to millions of numbers, with no dependencies between them. That is precisely the workload GPUs were built for: thousands of independent dot products computed simultaneously.

C = A × B          each element of C is an independent dot product

CPU:  computes elements a few at a time (with SIMD)
GPU:  computes thousands of elements in parallel

This is why a training job that takes weeks on a CPU can finish in hours on a GPU. One caveat worth knowing as a principal-level intuition: these workloads are often memory-bandwidth bound, not compute-bound — feeding data to all those cores fast enough is frequently the real bottleneck, which is why GPU memory bandwidth and batching matter so much. And Amdahl's law sets the ceiling: the speedup from parallelism is limited by the fraction of work that must remain sequential.

Performance work has one iron rule: measure first. Intuition about where a program spends its time is wrong far more often than right, and optimizing the wrong thing wastes effort while adding complexity. This chapter covers the numbers, the tools, and the laws that bound what's possible.

Latency Numbers Every Programmer Should Know

The single most useful mental model in performance is the relative cost of operations. They span more than eight orders of magnitude:

Key takeaway: the gap between cache and disk — or memory and network — is the difference between grabbing something off your desk and driving across the country for it. Good design minimizes trips to "distant" storage.

Profiling

"Premature optimization is the root of all evil." — Donald Knuth

Before changing anything, profile to find the actual bottleneck. Profilers come in flavors — CPU (where is time spent?), memory (where are allocations?), I/O (where do we wait on disk/network?), and lock (where do threads contend?). The CPU profile is often visualized as a flame graph:

The Optimization Hierarchy

Optimizations are not equal — apply them in order of impact:

1. Algorithmic — the biggest lever. Replacing an O(n²) with an O(n log n) on a million items cuts a trillion operations to twenty million. Always fix the algorithm before micro-optimizing.
2. Caching — a 95% hit rate against a 100 ms database query yields an average of 0.95 × 1ms + 0.05 × 100ms ≈ 6 ms, a ~17× win.
3. Batching — turn 1,000 individual inserts (1,000 round trips) into one batch insert (one round trip).
4. Connection pooling — reuse connections instead of paying ~50 ms to open one per request.
5. Compression — gzip a 1 MB response down to ~100 KB before sending it.
6. Parallel I/O — fetch independent resources concurrently so total time is the slowest one, not the sum.

Amdahl's Law — The Limit of Parallelism

Adding cores only speeds up the parallelizable part. The serial fraction caps your maximum speedup, no matter how many cores you throw at it: speedup = 1 / ((1 − P) + P/N).

Parallelizable	2 cores	8 cores	∞ cores
P = 50%	1.3×	1.8×	2×
P = 90%	1.8×	4.7×	10×
P = 99%	2.0×	7.5×	100×

Analogy: Nine women can't produce a baby in one month. The serial part (gestation) can't be parallelized — Amdahl's Law quantifies exactly that limit.

Little's Law — Sizing a System

A fundamental truth of any queue: L = λ × W — the average number of items in the system equals the arrival rate times the time each spends there. If a server handles 100 requests/second and each takes 0.5 s, then ~50 requests are in flight at any moment — so you need at least 50 threads or connections to keep up.

Analogy: A store with 60 arrivals/hour, each staying 30 minutes, always holds about 30 shoppers. To reduce crowding, either slow arrivals or speed up shopping.

For most of this book, "the computer" has been a server in a data center. But computing is increasingly pushed outward — to CDN edges, cell towers, gateways, and the billions of small devices that make up the Internet of Things. The edge is where physics (the speed of light) and economics (bandwidth) force computation closer to where data is born.

Edge Computing

The motivation is latency. A round trip to a distant cloud region is ~200 ms; a round trip to a nearby edge node can be ~5 ms — and some applications simply cannot wait:

Beyond latency, three other forces push work to the edge: bandwidth (a security camera generates terabytes a day — you can't ship it all to the cloud), privacy (process medical or smart-home data locally), and reliability (keep working when the cloud link drops). Real examples: Cloudflare Workers run JavaScript at 300+ locations, smart cameras run ML inference on-device, and self-driving cars do all safety-critical processing on board — they can't wait for a server.

Analogy: Cloud computing is ordering everything from one central warehouse — reliable, but slow delivery. Edge computing adds mini-warehouses in every neighborhood — fast for common items, with the central warehouse still behind them for everything else.

IoT Architecture

The Internet of Things connects vast numbers of constrained devices, and its architecture is a three-tier funnel:

MQTT

The protocols at the device tier are built for severe constraints — tiny payloads, intermittent power, flaky links. The dominant one is MQTT, a lightweight publish-subscribe protocol. Devices publish to a hierarchical topic (home/bedroom/temperature → 22.5), and any interested subscriber receives it via a broker — with three quality-of-service levels trading reliability against overhead:

• QoS 0 — at most once ("fire and forget")
• QoS 1 — at least once (acknowledged, may duplicate)
• QoS 2 — exactly once (guaranteed, but slowest)

Analogy: MQTT is a radio station. Sensors broadcast (publish) on frequencies (topics); interested listeners (subscribers) tune in; and the broker is the tower relaying the signal between them.

"It works on my machine" is the oldest complaint in software. Different environments — your laptop, the CI server, production — have different OS versions, libraries, and configuration, and code that runs in one breaks mysteriously in another. Virtualization and containers solve this by packaging the environment with the code, so what runs in development is bit-for-bit what runs in production.

Virtual Machines

A hypervisor carves one physical machine into several virtual ones, each running a complete guest operating system with its own kernel. The isolation is strong, but the cost is heavy — every VM duplicates an entire OS.

What Makes a Container

A container achieves isolation without a second OS by leaning on two Linux kernel features. This is the key insight: containers are just ordinary processes that the kernel has fenced off — there's no guest OS at all.

• Namespaces isolate what a process can see. A PID namespace shows it only its own process tree; a network namespace gives it its own IP and ports; mount and user namespaces give it their own filesystem view and user IDs.
• Cgroups (control groups) limit what a process can use — CPU, memory, I/O, and network bandwidth.

Analogy: Namespaces are blinders on a horse — it sees only its own lane. Cgroups are a governor on an engine — it can draw only so much power. Together they make a process believe it has the machine to itself while the kernel quietly keeps it boxed in.

Docker

Docker made containers usable. You describe an environment in a Dockerfile, build it into an image, and run the image as a container:

FROM python:3.11-slim              # base image
WORKDIR /app                       # working directory
COPY requirements.txt .            # copy dependency manifest
RUN pip install -r requirements.txt # install dependencies
COPY . .                           # copy application code
EXPOSE 8080                        # document the port
CMD ["python", "app.py"]           # default command

The relationship mirrors OOP: an image is a read-only template (a class); a container is a running instance of it (an object). You docker build an image, docker run a container, and docker push the image to a registry (like Docker Hub) for others to pull.

Crucially, each Dockerfile instruction creates a cached layer, and layers are shared between images:

Only the layers that changed are rebuilt — so editing your app code rebuilds the last layer while the expensive pip install layer is reused. This is also why ordering matters: put rarely-changing steps first.

Analogy: Layers are transparencies on an overhead projector — each adds something on top of the last. Change only the top sheet and you replace just that one, not the whole stack.

Orchestration with Kubernetes

One container is easy. Hundreds of containers across dozens of machines need an orchestrator. Kubernetes is the standard: you declare the desired state ("run 5 copies of this service") and it continuously works to make reality match.

That last idea — a control loop constantly reconciling desired state against actual state — is the heart of Kubernetes, and it delivers scheduling, auto-scaling, self-healing (restart failed containers, replace dead nodes), service discovery, rolling updates with zero downtime, and load balancing.

Analogy: Kubernetes is an airport operations manager — deciding which plane (container) goes to which gate (node), handling cancellations (restarting failures), managing traffic (load balancing), and scaling flights up or down with passenger demand.

Once a system is distributed across many services and machines, you can no longer attach a debugger and step through it. You have to understand it from the outside, by the signals it emits. That's observability — and it's worth distinguishing from plain monitoring. Monitoring watches for known failure modes ("is CPU above 90%?"). Observability is the richer goal: emitting enough data that you can ask new questions after the fact and diagnose problems you never anticipated — the "unknown unknowns."

The Three Pillars

Logs are timestamped records of discrete events — the narrative of what happened:

2024-01-15 14:23:01 INFO  [user-service]  User alice logged in
2024-01-15 14:23:02 ERROR [order-service] Order #1234 failed: DB timeout

Metrics are numerical measurements aggregated over time — cheap to store and ideal for dashboards and alerts:

http_requests_total{method="GET", status="200"}  150432
http_request_duration_seconds{quantile="0.99"}    0.250
cpu_usage_percent                                  73.2

Traces follow a single request as it fans out across services — the pillar that makes distributed bottlenecks visible:

Analogy: Logs are a diary ("today I bought milk"). Metrics are a fitness tracker (steps, heart rate, sleep — numbers over time). Traces are the GPS breadcrumbs of one specific journey — every turn, every stop, and how long each took.

SLIs, SLOs, and SLAs

Reliability needs to be measurable and agreed-upon. Three related terms — easy to mix up — make it concrete:

Term	Meaning	Example
SLI (Indicator)	The metric you actually measure	99.2% of requests finish in <200 ms
SLO (Objective)	Your internal target	99.5% should finish in <200 ms
SLA (Agreement)	A contractual promise to customers	99.9% uptime, or we issue refunds

The powerful idea that falls out of this is the error budget. If your SLO is 99.9%, you're allowed 0.1% unreliability — about 43 minutes per month. That budget isn't failure; it's a resource you deliberately spend on deploys, experiments, and migrations. Plenty of budget left? Ship faster. Budget exhausted? Freeze changes and shore up stability.

Analogy: The SLI is your current GPA, the SLO is the GPA you're aiming for, and the SLA is the GPA your scholarship requires. The error budget is how many points you can afford to drop and still keep the scholarship.

Everything fails eventually — disks, networks, servers, whole data centers. Reliability engineering is the discipline of building systems that keep working anyway. The mindset shift is to stop trying to prevent all failure (impossible) and instead design to tolerate and recover from it.

Failure Modes

Failures get harder to handle as they get less honest:

Mode	What happens	Difficulty
Crash	Process stops completely	Easy — detect via heartbeat timeout
Omission	Fails to send/receive some messages	Medium — looks like slowness
Timing	Response arrives too late	Hard — is it slow, or dead?
Byzantine	Behaves arbitrarily, even maliciously	Hardest — must out-vote liars

Most systems defend only against crash and omission failures; Byzantine tolerance is expensive and reserved for blockchains and aerospace.

Chaos Engineering

If failures are inevitable, the worst time to discover your weaknesses is during a real outage. Chaos engineering, pioneered by Netflix, flips this: deliberately inject failures in production, regularly, and fix what breaks. Netflix's Chaos Monkey kills random instances; Chaos Kong takes out entire regions. The method is disciplined — define normal behavior ("steady state"), hypothesize it will survive a given failure, inject that failure, and observe whether the hypothesis held.

Analogy: Chaos engineering is a fire drill. You don't wait for a real fire to discover the exits are blocked — you practice, find the problems, and fix them before it matters.

Redundancy

Tolerating failure means having spares, arranged in one of a few patterns:

Availability Math

Redundancy isn't just intuition — it's arithmetic. Availability is usually quoted in "nines":

Availability	Downtime per year
99% (two 9s)	3.65 days
99.9% (three 9s)	8.8 hours
99.99% (four 9s)	53 minutes
99.999% (five 9s)	5.3 minutes

The crucial insight is how components combine:

Analogy: Serial dependencies are Christmas lights wired in series — one bulb fails and the whole string goes dark. Parallel redundancy is lights wired in parallel — one fails, the rest stay lit.

Backpressure

A subtle but vital reliability mechanism: when a fast producer overwhelms a slow consumer, an unbounded buffer between them fills until it overflows — data loss or an out-of-memory crash. Backpressure lets the overloaded component push back upstream:

Analogy: Backpressure is a sink. If water comes in faster than it drains, it overflows. Backpressure either turns down the faucet (throttle the producer) or opens the drain wider (scale up consumers).

Most of this book assumes a world of abundant CPU, gigabytes of RAM, and the freedom to "just add a server." An enormous category of computing lives under the opposite constraints — where timing is a hard guarantee and memory is measured in kilobytes. This is the world of real-time and embedded systems, and it runs your car, your pacemaker, and the spacecraft leaving the solar system.

Real-Time Operating Systems

A general-purpose OS tries to be fast on average. A real-time OS makes a stronger promise: it guarantees deadlines. "I will handle this interrupt within 10 microseconds" — not usually, always.

The distinction splits into two flavors. Hard real-time means a missed deadline is a system failure — an airbag that fires late, a pacemaker that skips a beat, anti-lock brakes that hesitate. Soft real-time means a missed deadline merely degrades quality — a dropped video frame or a stutter in audio. Hard real-time forbids anything with unpredictable timing: no garbage collection, no paging to disk, no unbounded loops.

Analogy: A general-purpose OS is a restaurant promising "your food will be out soon." An RTOS is a factory assembly line where every step must complete within exact time bounds, or the product is defective.

Examples: FreeRTOS, VxWorks, and QNX — found in medical devices, cars, and spacecraft.

Embedded Systems

An embedded system is a computer inside something that isn't thought of as a computer — a microwave, a thermostat, a traffic light. The whole machine is often a single microcontroller:

The constraints are severe: clock speeds in megahertz (not gigahertz), memory in kilobytes (not gigabytes), strict power budgets (battery or solar), and a reliability bar where "just reboot it" isn't an option for a pacemaker. Frequently there's no operating system at all — you program "bare metal," directly against the hardware.

Interrupt-Driven Programming

With so little CPU to spare, you can't waste cycles constantly checking whether something happened (polling). Instead you register an interrupt — the hardware calls your code only when an event actually occurs, and the CPU sleeps in between:

// Polling — wastes every cycle checking:
while (1) { if (button_pressed()) do_something(); }

// Interrupt-driven — runs only when it matters:
void button_ISR() {   // Interrupt Service Routine
    do_something();    // called automatically on button press
}

Analogy: Embedded programming is living in a tiny apartment on a strict budget. Every byte of RAM and every CPU cycle matters; you cannot afford waste. It's the polar opposite of cloud computing.

Theory clicks into place when you assemble the pieces. Let's design a simplified URL shortener (think bit.ly) — small enough to hold in your head, yet it exercises almost every building block from this part. This is also the canonical systems-design interview question, so the reasoning here transfers directly.

Requirements

Always start by separating what it must do from how well it must do it — functional vs non-functional requirements:

• Functional: shorten a long URL into a short code (short.ly/abc123); redirect a short code back to the original.
• Non-functional: scale to billions of URLs; redirect latency under ~100 ms; high availability.

High-Level Design

Notice this picture is built entirely from earlier concepts: a load balancer, stateless app servers, a cache, a sharded database, and an ID generator. Systems design is mostly composition, not invention.

Walkthrough

Shortening a URL is a write — relatively rare:

POST /shorten   { "url": "https://very-long-url.com/article/..." }

1. Generate a unique ID (a distributed ID generator like Snowflake)
2. Encode it in base62 → "abc123"   (62^6 ≈ 56 billion codes)
3. Store code → URL in the database
4. Populate the cache
5. Return "https://short.ly/abc123"

Redirecting is a read — and the overwhelmingly common operation:

GET /abc123

1. Look up "abc123" in the cache (Redis)   → hit? return the URL
2. On miss, query the database             → found? cache it, return
3. Not found?                              → 404
4. Respond with 301 (permanent) or 302 (temporary) redirect

Scaling Decisions

This is where the analysis pays off. A URL shortener is overwhelmingly read-heavy — roughly 100 reads per write — which drives every choice:

• Cache aggressively. URL popularity follows a power law: a few links get most of the traffic. A cache covering the hot set absorbs the vast majority of reads.
• Shard the database by the short code's hash, so writes and storage spread across machines.
• Generate IDs without collisions across servers using a distributed scheme (Twitter's Snowflake), so no two app servers ever mint the same code.
• Add read replicas to scale reads beyond what the primary can serve.

A Checklist for Any System

The same skeleton applies whether you're designing a URL shortener, a chat app, or a payment system. Walk it top to bottom:

1. Requirements — functional and non-functional
2. Capacity estimation — requests/sec, storage, bandwidth
3. API design — endpoints, methods, payloads
4. Data model — schema and access patterns
5. High-level design — components and how data flows between them
6. Detailed design — specific algorithms and data stores
7. Scaling — horizontal scaling, caching, sharding
8. Reliability — replication, failover, backups
9. Observability — metrics, logging, alerting
10. Security — authentication, encryption, rate limiting

The art isn't memorizing this list — it's knowing, for the system in front of you, which steps carry the real risk and deserve the most attention. That judgment is what separates a junior engineer who can implement a design from a principal who can create one.