DMA and Modern Buses: PCIe and USB

Think about what a modern SSD delivers: several gigabytes every second. If the processor had to fetch each byte from the disk controller and store it in memory, the way our keyboard handler did, it would do nothing else, and still couldn't keep up. Copying data is a job for a machine that does nothing but copy.

Moving data by hand

So far, every byte that came from a device passed through the processor: read from a device register, then stored to memory. That's called programmed I/O. x86 even has a special instruction for it, rep insw, which reads a whole block of 16-bit words from one port into memory. PC hard disks used it for years to read each 512-byte sector:

        mov     dx, 0x1F0       ; the hard disk's data port
        mov     ecx, 256        ; 256 words = 512 bytes
        rep insw                ; read them all into memory at [rdi]

It's one instruction, but the processor still does all the work, one word at a time, at the speed of the slow device. And if each piece of data arrives with its own interrupt, the cost of handling the interrupts quickly swamps everything else.

Direct memory access

The answer is direct memory access, or DMA. A separate circuit, a DMA controller, moves the data between the device and memory over the bus, without the processor. The processor only has to set up the transfer, by writing a few device registers:

1. The memory address where the data should go (or come from).
2. How many bytes to move.
3. Which direction: from the device to memory, or the other way.

Then it goes back to its own work. The DMA controller copies each piece as the device produces it and, when everything has been moved, raises a single interrupt: "transfer complete".

Both sectors arrive at exactly the same moment: the disk itself sets the pace. The difference is what the processor was doing meanwhile. With DMA, it spent nearly all its time on real work. The only thing it shares with the DMA controller is the bus: when both want memory at the same instant, one has to wait a moment. That's why DMA is sometimes said to "steal cycles".

The IBM PC had a DMA controller from day one, an Intel 8237 with four channels at ports 0x00 to 0x0F (it's the dma1 in Chapter 11's port list). Channel 2 served the floppy drive. And channel 0 had a surprising job: every 15 microseconds, prompted by the timer chip, it did a dummy read that refreshed the DRAM of Chapter 8.

Devices that copy for themselves

Today there's no central DMA controller doing the copying. Every fast device has its own DMA engine built in and can read and write memory by itself, called bus mastering. The driver no longer sets up one transfer at a time. Instead, it prepares a list of buffers in memory, arranged as a ring:

• Each entry, a descriptor, says "here is a free buffer at this address, this big".
• The device fills buffers in order, marks each one as done, and moves on, wrapping around at the end.
• The driver takes the finished buffers from behind, hands back empty ones, and the two chase each other around the ring.

A network card works exactly like this: packets land in memory without the processor touching them. And since the device can fill several buffers before anyone looks, it doesn't need to interrupt for every single one. It can wait until a few have collected, called interrupt coalescing.

When packets come in faster than the driver can take them out, the ring fills and the card has nowhere to put new ones, so it drops them. Coalescing helps, because the processor pays the interrupt overhead once per batch instead of once per packet. Real network drivers go further still: under heavy load they switch interrupts off completely and poll the ring in a loop. Polling, it turns out, is the right answer when there's always something there!

NVMe SSDs take the ring idea to the extreme. The driver puts commands in a submission queue, the drive puts results in a completion queue, and a drive can offer up to 65,535 such queue pairs, so every processor core can have its own.

From shared wires to PCI Express

Every one of these devices needs a path to the processor and memory: a bus. The IBM PC's expansion bus, later called ISA, was simply the processor's own wires brought out to five slots: 8 data lines, 20 address lines, plus the control lines like "memory read" and "I/O write" from Chapter 11. It ran at the processor's 4.77 MHz and could move roughly 1 MB per second. Every card saw every signal, and only one could talk at a time.

As processors got faster, buses got wider and quicker. PCI (1992) was 32 bits wide at 33 MHz: 133 MB per second, shared by all the cards. But pushing parallel wires further ran into a wall of physics. At high speeds, signals on 32 neighboring wires no longer arrive at exactly the same time, and the receiver can't tell which bits belong together.

The solution was surprising: use fewer wires, but run each one far faster. PCI Express (PCIe, 2003) connects each device by its own private link, made of one or more lanes. A lane is just two pairs of wires, one pair for each direction, sending bits one after another at billions per second. Since each lane recovers its own timing from the bits it receives, there's no problem keeping wires in step. Need more speed? Use more lanes: x1, x4, x8 or x16. Data travels in packets, routed through switches, much like a tiny network inside the computer.

Every generation has doubled the speed per lane. The first two lost 20% of the bits to their line code, which makes sure the signal changes often enough for the receiver to keep time; later ones found cleverer codes. The PCIe 7.0 specification, published in 2025, doubles the speed yet again.

Finding the devices

Remember the jumpers of the IBM PC, where the user had to choose each card's ports and IRQ by hand? PCI ended that. Every PCI device has a small block of configuration space that the firmware and the operating system can read at boot. At its very start are two numbers: a vendor ID and a device ID. Intel's vendor ID is 8086, a nod to its famous processor.

At boot, the system checks every possible position (bus, device and function), reads the IDs, and loads the matching driver. It then hands each device the memory ranges for its registers. To find out how much space a device needs, it writes all 1s into the device's address register and reads it back: the bits the device won't let it set reveal the size. Then it writes in the real address. That's how the memory-mapped I/O of Chapter 11 gets its addresses without anyone setting a jumper.

USB: one plug for everything

Outside the box, the same story played out. In the 1990s a PC had a different plug for each kind of device: keyboard, mouse, printer, joystick, modem. USB (Universal Serial Bus, 1996) replaced them all with one.

• The host is in charge. Devices never speak unless asked. The computer's host controller asks each device in turn, many times a second, whether it has anything to say.
• A tree of hubs. Up to 127 devices can hang off one controller, through hubs.
• Devices describe themselves. When you plug one in, the host gives it an address and asks for its descriptors: vendor and product ID, and its class. Every keyboard belongs to the HID class (human interface device), so one driver works for all of them.

Wait: the host polls the keyboard? Yes, typically every few milliseconds. But it's the host controller hardware that does the asking, not the processor. The controller writes the answers into memory by DMA, using rings of descriptors, and interrupts the processor only when there's news. All the techniques of this part of the book, working together.

USB has kept speeding up: 12 Mb/s in 1996, 480 Mb/s with USB 2.0 (2000), 5 Gb/s with USB 3.0 (2008), and up to 80 Gb/s with USB4 version 2.0. USB4 can even carry PCI Express and video signals through the same cable.

Meanwhile, inside a real CPU

The processor's side of all this is small. rep insw is three bytes: f3 66 6d. For DMA, the processor only writes a few device registers and later handles one interrupt. RISC-V has no port instructions at all, so on RISC-V systems almost every device does its own DMA.

On Linux, lspci lists the PCI devices. Here are some from the computer this book was written on. The numbers in brackets at the end are the vendor and device IDs:

$ lspci -nn                          (trimmed)
00:02.0 VGA compatible controller: Intel AlderLake-S GT1 [8086:4680]
00:14.0 USB controller: Intel ... USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0]
00:17.0 SATA controller: Intel ... SATA Controller [AHCI Mode] [8086:7ae2]
00:1f.0 ISA bridge: Intel Z690 Chipset LPC/eSPI Controller [8086:7a84]
04:00.0 Ethernet controller: Realtek RTL8125 2.5GbE Controller [10ec:8125]

Notice the "ISA bridge" in a 2021 chipset. There are no ISA slots any more, but the chip still answers at the old ports so that the keyboard controller and timer of Chapters 11 and 12 keep working. The Realtek network chip sits on a PCIe 2.0 x1 link: 500 MB/s, comfortably more than the 312.5 MB/s that 2.5 Gb Ethernet needs. And lsusb -t shows the USB tree, including the keyboard this was typed on:

$ lsusb -t                           (trimmed)
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 010: Dev 003, If 0, Class=Human Interface Device, Driver=usbhid, 12M   ← keyboard
    |__ Port 011: Dev 004, If 0, Class=Audio, Driver=snd-usb-audio, 12M
    |__ Port 013: Dev 006, If 0, Class=Hub, Driver=hub/4p, 480M
    |__ Port 014: Dev 007, If 0, Class=Wireless, Driver=btusb, 12M                  ← Bluetooth
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/9p, 20000M/x2

The keyboard still runs at 12 Mb/s, USB's 1996 "full speed": plenty for a few keys.

Try it yourself

1. A device delivers a 4 KB page, one byte at a time. With programmed I/O and one interrupt per byte, how many interrupts does that take? And with DMA?
   Show answer

   4,096 interrupts with programmed I/O, one per byte. With DMA, just one: "transfer complete".
2. PCIe 1.0 sends 2.5 billion bits per second on each lane, using the 8b/10b code. How many bytes of real data per second is that, per lane and direction?
   Show answer

   Only 8 of every 10 bits are data: 2.5 × 8/10 = 2 billion data bits per second. Divided by 8, that's 250 MB/s.
3. Could a PCIe 4.0 x4 link carry a 7 GB/s SSD?
   Show answer

   Yes, just about. Each PCIe 4.0 lane carries about 1.97 GB/s, so four lanes give about 7.9 GB/s. That's why the fastest PCIe 4.0 SSDs top out at around 7 GB/s.
4. In the ring demo at 0.6 packets per tick, why does "interrupt per packet" drop packets while "wait for a few" doesn't?
   Show answer

   With an interrupt per packet, each packet costs 2 ticks of processor time (1 of overhead, 1 of processing), so the driver can handle at most 0.5 packets per tick. With batches of 4, each batch costs 1 + 4 = 5 ticks for 4 packets: up to 0.8 per tick, enough to keep up.
5. A USB keyboard is polled every few milliseconds. Isn't that the wasteful polling of Chapter 11 all over again?
   Show answer

   No, because the processor isn't the one polling. The USB host controller asks the keyboard in hardware, writes any answer into memory by DMA, and interrupts the processor only when a key report has actually arrived.

What we learned

• With programmed I/O, the processor moves every byte itself. DMA lets a device move data to and from memory on its own, with one interrupt at the end.
• Modern devices are bus masters that work through rings of descriptors; interrupt coalescing cuts the interrupt overhead; an IOMMU keeps devices in their place.
• Parallel shared buses (ISA, PCI) gave way to PCI Express: private serial lanes carrying packets, doubling in speed each generation.
• PCI devices identify themselves through configuration space (vendor and device IDs) and receive their addresses automatically at boot.
• USB is host-controlled, with hubs, self-describing devices and classes, from 12 Mb/s to 80 Gb/s.

That completes Part IV. Our computer can now talk to the world: through ports and memory-mapped registers, by interrupts, and by DMA across PCI Express and USB. In Part V we look at one device in depth, the one that remembers everything when the power goes off. Chapter 14 goes from spinning disks to SSDs.