Part IX · Chapter 27

Networking: From Bits on a Wire to the Web

You're reading this page because, a moment ago, your computer sent a few small packets across the world and a server sent the page back. On the way, your request crossed cables, radio waves and glass fibres, and passed through a dozen machines that knew nothing about you or this book. This chapter follows that journey, layer by layer.

A network is just computers sending bits to each other. The trick is doing it reliably, across millions of computers built by different companies, over links that are sometimes slow, sometimes busy, and sometimes broken. The internet manages this by splitting the problem into layers, each solving one part, and each relying on the layer below.

Bits on a wire

At the bottom, bits have to become something physical. On a copper network cable, they're voltages, just like the signals inside a computer (Chapter 1), but sent much farther. Gigabit Ethernet, the kind of connection the computer this book was written on uses, sends on all four pairs of wires in the cable at once, using five different voltage levels, to squeeze a billion bits through every second. Each bit lasts about a nanosecond. In optical fibre, bits are flashes of laser light in a glass thread thinner than a hair, which can cross oceans. In Wi-Fi and mobile networks, they're changes in a radio wave.

The network card turns bits into signals and back. To the computer it's a device like any other: it has registers, raises interrupts, and copies data to and from memory by DMA, using the rings of descriptors we saw in Chapter 13. On this computer it's the Realtek chip whose driver, r8169, we met in Chapter 19.

Layers and packets

Data doesn't flow as one long stream. It's chopped into packets of up to about 1,500 bytes, and each one travels on its own. Every layer of the network wraps the packet in its own header, like putting a letter in an envelope, then that envelope in a bigger one:

Application: what the programs actually say to each other, like a web browser's request.
Transport (TCP or UDP): which program at each end, using port numbers, and, for TCP, getting everything there complete and in order.
Network (IP): which computer, anywhere in the world, using IP addresses.
Link (Ethernet or Wi-Fi): getting across one cable or radio link to the next box, using MAC addresses, the hardware addresses built into every network card.

Wrap a real web request in each layer, one at a time. Watch the headers pile up around the data.

The beauty of layers is that each one can change without the others noticing. Your browser doesn't care whether the packets go by cable or Wi-Fi, and the routers in between don't care whether they carry a web page, an email or a video call.

Addresses and routing

An IPv4 address is 32 bits, written as four numbers from 0 to 255: the web server for this book is 54.38.118.204. That allows about 4.3 billion addresses, which the world has run out of. So most homes share one public address between all their devices: the home router gives each device a private address, like 192.168.13.35, and swaps it for the public one on the way out, a trick called NAT (network address translation). The long-term fix is IPv6, with 128-bit addresses: enough for every grain of sand on Earth to have billions. This book's server has one of those too.

No computer knows the whole internet. A router is a computer that connects several networks and passes packets between them. Each router has a routing table: a list of address ranges, each with the neighbour to send them to. A range is written as an address and a prefix length, like 54.0.0.0/8, meaning "every address whose first 8 bits match 54". When several lines match, the router picks the most specific one. A packet crosses the internet one hop at a time, each router knowing only the next step.

Send a packet and watch each router make its choice. Then run traceroute to discover the path. (An illustrative network, with real addresses at the two ends.)

Reliable delivery: TCP

IP makes no promises. Packets can be lost when a router is too busy, arrive out of order when they take different paths, or even arrive twice. For a web page or a file, that's not good enough, so most programs use TCP (Transmission Control Protocol) on top. TCP gives each program the illusion of a perfect, two-way stream of bytes:

A handshake sets up the connection first.
Every byte gets a sequence number, so the receiver can put segments back in order.
The receiver sends acknowledgements back. Anything not acknowledged in time is sent again.
The sender keeps several segments in flight at once, up to a window, so it isn't waiting for a round trip after every packet. If packets start getting lost, TCP takes it as a sign the network is overloaded and slows down: congestion control, which keeps the internet from jamming up.

Play the conversation, then tick "Lose segment 3" and play it again. Watch how TCP notices and repairs the loss.

Sometimes speed matters more than perfection. UDP (User Datagram Protocol) just sends packets, with port numbers and a checksum, and nothing more. Video calls and games use it, since a late packet is useless anyway, and so does DNS, where one short question gets one short answer: if either is lost, just ask again.

Names and the web

People remember names, not numbers. The Domain Name System (DNS) turns a name like downtothemetal.dev into an IP address. It's a worldwide, layered directory. Thirteen named sets of root servers know who is responsible for each ending, like .dev; the .dev servers know which name servers hold the records for downtothemetal.dev; and those servers know the address. Answers are cached along the way, so most lookups take a millisecond or two.

Then the browser speaks HTTP (HyperText Transfer Protocol). In its classic form, it's refreshingly simple: plain lines of text, a request like GET / HTTP/1.1 followed by some headers, and a response starting with a status like 200 OK, more headers, and the page itself. Today almost everything uses HTTPS: the same HTTP, encrypted with TLS, so nobody along the way can read or change it, and with a certificate proving that the server really is who it claims to be. Newer versions of HTTP send several requests at once over one connection, and HTTP/3 even replaces TCP with a new protocol, QUIC, built on UDP.

Meanwhile, inside a real CPU

Here's how the computer this book was written on is connected: one wired Ethernet card, at 1,000 megabits per second, with a private address on the home network, and a routing table whose default route sends everything else to the home router:

$ ip -br addr show enp4s0
enp4s0     UP     192.168.13.35/24
$ cat /sys/class/net/enp4s0/speed /sys/class/net/enp4s0/mtu
1000                                 ← megabits per second
1500                                 ← the largest packet, in bytes
$ ip route                           (trimmed: the routes for virtual machines are left out)
default via 192.168.13.1 dev enp4s0 proto dhcp src 192.168.13.35
192.168.13.0/24 dev enp4s0 proto kernel scope link src 192.168.13.35

The kernel counts every packet. In the 11.9 days since the computer was switched on:

$ cat /proc/net/dev                  (enp4s0 only, rearranged)
received: 75,090,212,683 bytes in 61,782,381 packets
sent:      4,961,938,530 bytes in 28,910,568 packets

Looking up this book's server, the long way round: dig +trace skips the local cache and asks a root server, then the .dev servers, then the domain's own name servers:

$ dig +trace downtothemetal.dev        (trimmed)
.                   NS  a.root-servers.net. … m.root-servers.net.
dev.                NS  ns-tld1.charlestonroadregistry.com. …  ← from c.root-servers.net, 16 ms
downtothemetal.dev. NS  ns1.dnsimple-edge.com. …               ← from ns-tld1.charlestonroadregistry.com, 56 ms
downtothemetal.dev. A   54.38.118.204                          ← from ns3.dnsimple-edge.io, 95 ms

With the answer cached, the same lookup took 1 millisecond. Tracing the path to the server found the home router, then a second router on the same premises, and after that silence: many routers on the internet are set up not to answer traceroute's probes.

$ tracepath -n 54.38.118.204
 1:  192.168.13.1      0.211ms
 2:  192.168.2.1       3.471ms
 3:  no reply
 …

Finally, fetching this book's home page, and timing each step. Each of the first three steps costs about one round trip, roughly 90 ms, across the ocean of routers in between:

$ curl -w … https://downtothemetal.dev/
DNS lookup done:      0.001 s    ← cached
TCP connected:        0.088 s    ← the handshake: one round trip
TLS secured:          0.183 s    ← encryption agreed: one more
first byte of page:   0.274 s    ← request sent, answer starts arriving
whole page:           0.356 s    ← 11,064 bytes, HTTP/2, status 200

And the conversation itself, using the plain HTTP/1.1 text format (after TLS has decrypted it):

> GET / HTTP/1.1
> Host: downtothemetal.dev
> User-Agent: curl/8.5.0
> Accept: */*

< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000     ← "I also speak HTTP/3"
< Cache-Control: no-cache
< Content-Type: text/html; charset=utf-8
< Server: nginx

Try it yourself

At 1 gigabit per second, how long does it take to send a full Ethernet frame of 1,518 bytes?
Show answer
1,518 × 8 = 12,144 bits. At 1,000,000,000 bits per second, that's 12.1 microseconds.
In the layers demo, a 57-byte request needs 58 bytes of headers, more than the request itself. What fraction is headers when a full packet carries 1,460 bytes of data?
Show answer
The headers are the same 58 bytes: 58 ÷ (1,460 + 58) ≈ 3.8%. Small packets are expensive; big ones are efficient.
The address 54.38.118.204 matches both 54.0.0.0/8 and 54.32.0.0/11. Which line does a router use?
Show answer
54.32.0.0/11, because it's more specific: it matches 11 bits instead of 8. Routers always use the longest matching prefix.
Why did fetching the page take at least three round trips?
Show answer
One for TCP's handshake, one for TLS to agree on encryption keys, and one for the HTTP request and the start of the response. With a round trip of about 90 ms, that's at least 270 ms. This is why newer protocols like QUIC combine these steps.
DNS normally uses UDP, not TCP. Why does that make sense?
Show answer
A lookup is one small question and one small answer. TCP would need a whole round trip for its handshake first, doubling the time. If a UDP packet gets lost, the program just asks again.

What we learned

Bits travel as voltages on copper, light in fibre or radio waves; the network card converts them, using DMA like any device.
Data is split into packets, wrapped in a header by each layer: link (Ethernet), network (IP), transport (TCP/UDP) and application.
IP addresses identify computers; routers forward packets hop by hop using routing tables and the longest matching prefix.
TCP builds a reliable, ordered stream with handshakes, sequence numbers, acknowledgements and retransmission; UDP just sends.
DNS turns names into addresses; HTTP, secured by TLS, carries the web.

One more chapter to go. GPUs started out drawing pictures, but today they, and a new family of chips, do something else entirely: training and running artificial intelligence. Next: accelerators.