The Computer

Everything inside a computer is ultimately bits — yet we routinely store negative numbers, fractions, Chinese characters, and emoji. The bridge between raw bits and meaning is encoding: an agreed-upon convention for what a pattern of bits represents. This chapter covers the encodings every engineer eventually trips over.

Integers

Unsigned integers are the straightforward case — plain binary, where 8 bits cover 0 to 255. The interesting question is how to represent negative numbers, and the answer the whole industry settled on is two's complement, where the most significant bit carries the sign:

To negate a number, flip all its bits and add 1. The reason two's complement won is elegant: addition works identically for positive and negative values, so the CPU needs only one adder circuit:

 5 + (-3):
 00000101 + 11111101 = 00000010 = 2   ✓ (the carry out is discarded)

Analogy: Two's complement is an odometer rolling over. At 00000, subtracting 1 gives 99999 — which stands in for −1. The arithmetic just works with the wraparound.

Integer Overflow

That same wraparound is a hazard. When a value exceeds its type's range, it silently rolls over:

8-bit unsigned:  255 + 1 = 0      (wraps to the bottom)
8-bit signed:    127 + 1 = -128   (wraps to the most negative)

This has caused real disasters: the Ariane 5 rocket exploded in 1996 when a 64-bit float was forced into a 16-bit integer; Pac-Man's level 256 "kill screen" is an 8-bit counter overflowing; and YouTube once had to widen its view counter when "Gangnam Style" blew past 2³². Overflow bugs are silent until they aren't.

Floating Point (IEEE 754)

How do you store 3.14 or 0.000001 in binary? The same way science writes very large and very small numbers — scientific notation — split into a sign, an exponent, and a mantissa:

This buys enormous range at the cost of exactness. The famous gotcha:

>>> 0.1 + 0.2
0.30000000000000004     # not exactly 0.3!

The reason: 0.1 in binary is a repeating fraction (0.0001100110011…), just as 1/3 is 0.333… in decimal — it can't be stored exactly in a finite number of bits. There are also special values: ±Infinity, and NaN (Not a Number), which is famously not equal to itself by design.

Analogy: Floating point is a ruler with fixed tick marks. It spans a huge range, but you can only land on the marks — most real lengths fall between them and get rounded.

Never use floats for money. Rounding errors compound into real financial bugs. Use integer cents or a dedicated decimal type:

BAD:   $10.10 as a float  → may become 10.099999999
GOOD:  1010 cents as an int → always exact

Character Encoding

ASCII (1963) used 7 bits for 128 characters — enough for English, and nothing else. 'A' is 65, 'a' is 97, '0' is 48. But the world writes in 中文, العربية, and 🎉.

Unicode solves this by assigning every character in every script a unique number called a code point — 'A' is U+0041, '中' is U+4E2D, '🎉' is U+1F389 — over 150,000 and counting. Unicode is the catalog; it doesn't say how to store those numbers as bytes. That's the job of an encoding, and the one that won is UTF-8:

UTF-8 won because it's backward-compatible with ASCII (English text is unchanged), has no byte-order ambiguity, stays compact, and is self-synchronizing — you can jump into the middle of a stream and find character boundaries. The practical trap is that "length" is ambiguous: "🎉".length is 2 in JavaScript (UTF-16 code units), 1 in Python 3 (code points), and 4 if you count UTF-8 bytes. Conflating bytes, code points, and characters is the source of countless string bugs.

Endianness

A final low-level convention: when a multi-byte value sits in memory, which byte comes first?

x86/x64 CPUs are little-endian; network protocols standardized on big-endian ("network byte order"); ARM is configurable. Mismatches here corrupt data silently when it crosses between systems.

Analogy: It's how you write a date. Month/day/year puts a less-significant field first; ISO 8601's year-month-day puts the most-significant first. Same information, opposite ordering — and you'd better agree which one you're using.

The CPU is the only component in your computer that can execute instructions. Everything else — memory, disk, GPU, network — exists to feed it data or handle work it delegates.

Von Neumann Architecture

In 1945, John von Neumann described the architecture that nearly every computer since has followed:

The key insight: both instructions and data live in the same memory. Your running program is just bytes in RAM. The CPU fetches those bytes, figures out what each one means, and acts on it.

Analogy: The CPU is a chef in a kitchen. Registers are the countertop — a few inches of immediate workspace. The ALU is the chef's hands: the only part that does actual work. The Control Unit is the recipe card, directing each step. RAM is the pantry. Disk is the grocery store — huge capacity, but you don't make a store run mid-dish.

Inside the CPU

Three components make computation happen:

• Control Unit (CU): Reads and interprets each instruction, driving all other units.
• Arithmetic Logic Unit (ALU): All computation — addition, subtraction, comparisons, bitwise operations — happens here.
• Registers: A handful of ultra-fast storage cells built into the chip. A 64-bit CPU has 16–32 general-purpose registers. Accessing a register takes ~1 cycle; accessing RAM takes ~100–300 cycles.

On x86-64 you'll encounter most often: rax, rbx, rcx, rdx (general-purpose), rsp (stack pointer), rbp (frame pointer), rip (instruction pointer).

The Fetch-Decode-Execute Cycle

Every instruction your computer runs follows this loop:

1. Fetch: The Program Counter (rip on x86-64) holds the memory address of the next instruction. The CPU reads those bytes from memory.
2. Decode: The control unit interprets the binary — ADD? LOAD? JUMP?
3. Execute: The ALU operates, registers update, or memory reads/writes happen.
4. Advance PC: Move to the next instruction, unless a branch jumped elsewhere.

Analogy: Reading a recipe step by step. Read the step (fetch), understand it (decode), do it (execute), move your finger to the next one. A step that says "if the sauce is too thick, go back to step 3" is a conditional branch.

Instruction Set Architecture (ISA)

The ISA is the contract between hardware and software: the complete list of operations a CPU understands, how they're encoded in binary, and what happens to registers and memory when each executes.

Modern x86 CPUs are CISC on the outside — they accept complex multi-byte instructions — but internally translate them to simple RISC micro-operations ("µops") before execution. This is why x86 code can run fast despite a complex encoding.

Caches — Bridging the Speed Gap

Registers are fast but scarce. RAM is abundant but slow (~100–300 cycles per access). Between them sits a hierarchy of caches:

When the CPU needs data, it checks L1 first, then L2, then L3, then RAM. A cache hit costs a handful of cycles. A cache miss costs hundreds. This is why sequential memory access ("cache-friendly" code) can make inner loops 10–100× faster than random access — sequential reads let the hardware prefetcher load upcoming data before it's needed.

Pipelining

Without pipelining, every instruction must complete all its stages before the next begins. Pipelining overlaps them — while one instruction executes, the next decodes, and the one after fetches:

A 5-stage pipeline can sustain one instruction completing per clock cycle in the steady state, even though each takes 5 cycles end-to-end.

Three hazards break this overlap:

• Data hazard: Instruction B needs a result instruction A hasn't produced yet. Fixed by forwarding (routing the partial result directly to B) or inserting stall cycles.
• Control hazard: A branch — the next instruction to fetch is unknown until the branch resolves. Fixed by branch prediction: the CPU guesses and speculatively executes ahead. Modern CPUs predict correctly ~97% of the time; a miss flushes the pipeline and re-fetches from the correct path.
• Structural hazard: Two instructions need the same hardware unit simultaneously. Fixed by duplicating the resource or reordering.

Superscalar and Multi-Core

Modern CPUs squeeze more throughput through three more techniques:

• Superscalar: Multiple parallel pipelines — 2–6 instructions complete per cycle when they're independent.
• Out-of-order execution: Dynamically reorder instructions at runtime to keep pipelines full, preserving only the logical order of results.
• Multi-core: Multiple complete CPUs on one die, each with private L1/L2 caches, sharing L3 and RAM.

Analogy: Superscalar is one chef with four hands. Multi-core is four chefs. Both increase throughput — but four chefs must coordinate so they don't grab the same ingredient at once. That coordination problem is the source of every concurrency bug you'll ever debug.

High-level languages are conveniences for humans. The CPU understands only one thing: machine code — raw binary bytes encoding exact operations.

Machine Code

Every instruction is a sequence of bytes. In x86-64, this two-byte sequence:

B0 61

means "move the value 97 (0x61) into register al." B0 is the opcode for that specific MOV variant. 61 is the operand. There is no runtime interpreting these bytes — the CPU's decode hardware maps them directly to internal signals that drive the ALU and registers.

Your compiled binary is ultimately a file full of such bytes. When the OS loads it, those bytes land in memory and rip points at the first one.

Assembly Language

Assembly is the human-readable, one-to-one representation of machine code. Each mnemonic maps to exactly one machine instruction:

; x86-64 Linux: print "Hello" then exit
section .data
    msg db "Hello", 10        ; string + newline byte

section .text
global _start
_start:
    mov  rax, 1               ; syscall number: write
    mov  rdi, 1               ; fd: stdout
    mov  rsi, msg             ; pointer to buffer
    mov  rdx, 6               ; byte count
    syscall                   ; hand control to the OS

    mov  rax, 60              ; syscall number: exit
    xor  rdi, rdi             ; status code 0
    syscall

An assembler (NASM, GAS) translates each mnemonic to its byte encoding. A linker then joins multiple object files and resolves symbolic names into actual memory addresses.

Core Instruction Categories

Category	Examples	What they do
Data move	mov, push, pop	Copy values between registers and memory
Arithmetic	add, sub, imul, idiv	Integer math
Logic	and, or, xor, not, shl, shr	Bitwise operations
Comparison	cmp, test	Set flags without storing a result
Control flow	jmp, je, jne, jg, jl	Unconditional / conditional jump based on flags
Functions	call, ret	Push return address + jump / pop return address + jump back
System	syscall	Transition to kernel mode to request OS services

cmp a, b subtracts b from a and discards the result, but sets the flags register (zero, negative, overflow). The subsequent conditional jump reads those flags. This is how all branching works at the hardware level.

The Call Stack

When a function calls another, the CPU needs to remember where to return — and the called function needs space for its own local variables. Both live on the call stack:

The mechanics in sequence:

• call foo: pushes the return address (the instruction after call) onto the stack, decrements rsp, then jumps to foo.
• Prologue: the called function saves the caller's rbp, sets rbp = rsp, then decrements rsp further to reserve space for locals.
• ret: restores rsp and rbp, pops the return address back into rip.

Analogy: The stack is a pile of cafeteria trays. Each function call adds a tray on top with its local state. When the function returns, its tray is removed — the caller's tray is exactly as it was.

Stack overflow is what happens when recursion goes too deep: every call adds a frame, the stack exhausts its fixed region (8 MB by default on Linux), and the OS signals SIGSEGV.

Calling Conventions

When one function calls another, both sides must agree on how arguments and return values are passed. This contract is the calling convention (or ABI — Application Binary Interface).

On x86-64 System V (Linux, macOS):

• First 6 integer/pointer arguments: rdi, rsi, rdx, rcx, r8, r9
• Floating-point arguments: xmm0–xmm7
• Return value: rax
• Arguments beyond 6: pushed onto the stack in reverse order

The compiler generates exactly this layout. When you hit a crash and open a debugger, rdi holds the first argument to the crashing function — because that's the ABI.

Analogy: Assembly is writing a novel one character at a time, specifying every ink stroke. Total control, immense tedium. In practice you write it only to understand what your compiler is generating, or to hand-optimize a hot inner loop that the compiler can't see past.