Down to the Metal

Part V · Chapter 14

From Spinning Disks to SSDs

RAM forgets everything the moment the power goes off. To keep your files, photos and programs, a computer needs storage that remembers on its own. For fifty years that meant tiny magnets on spinning platters. Today it's mostly electrons trapped in flash memory. The two work in completely different ways, and both have surprising quirks.

Remember the leaky capacitors of Chapter 8? DRAM has to be refreshed many times a second, and when the power stops, its contents fade away within moments. Storage is the opposite: slow, but non-volatile. It keeps its data for years with no power at all.

Remembering with magnets

A hard disk stores bits as tiny magnetized regions on a spinning platter coated with magnetic material. A head at the end of a swinging arm writes bits by magnetizing the surface one way or the other, and reads them back by sensing where the magnetism changes direction. The head never touches the platter: it flies on a cushion of air just a few nanometers above it, closer than the size of a virus.

The IBM PC started with floppy disks, which work the same way but with the head pressed against a flexible plastic disk. A 5¼-inch floppy spun at 300 turns per minute and held 160 KB: 40 rings of 8 pieces of 512 bytes.

Tracks and sectors

The data on a platter is arranged in rings called tracks, and each track is divided into sectors: the smallest piece a disk can read or write. For decades a sector held 512 bytes; modern drives use 4,096. A drive usually has several platters, with a head for each surface, all moving together on one arm.

The original PC addressed a sector by its cylinder, head and sector numbers. Today the drive simply numbers all its sectors from 0 upwards, called logical block addressing (LBA), and works out for itself where each one physically lives.

To read a sector, three things must happen, one after another:

  1. Seek: swing the arm until the head is over the right track.
  2. Rotational wait: wait for the sector to spin round under the head.
  3. Transfer: read the sector as it passes.
A toy disk with 8 tracks of 12 sectors, spinning at 7,200 RPM like many real drives. The orange sectors are wanted. Serve them in arrival order, then with the elevator. Time is slowed down 80 times.

Why disks are slow

Only a tiny fraction of the time goes on actually reading. The rest is mechanics: the arm has to accelerate, stop and settle, and the platter turns only 120 times a second. On a typical 7,200 RPM drive, an average seek takes 8 or 9 milliseconds and the average rotational wait is half a turn, 4.2 ms. That allows only about 100 reads per second from random places on the disk. To a processor that runs billions of instructions a second, that's an eternity.

Yet the same drive can stream around 200 MB per second when it reads sectors one after another, with no seeking at all. So everyone works hard to avoid random access:

Flash: trapping electrons

An SSD (solid-state drive) has no moving parts at all. It stores bits in flash memory, built from a special kind of transistor: the switch of Chapter 1, with an extra gate added that is completely wrapped in insulation, a floating gate.

Apply a high voltage, and electrons are forced through the insulation into the floating gate. When the voltage is removed, they're trapped there, for years. The trapped charge makes the transistor harder to switch on, so the chip can read the cell by checking how hard it is to turn on. No charge reads as one value, charge as the other.

To store more bits in each cell, manufacturers store different amounts of charge and tell the levels apart. Two bits need four levels; four bits need sixteen. And to fit more cells, modern chips stack them in layers, more than 200 of them, like a skyscraper of transistors.

Pick how many bits each cell stores, then click a level to program it. Each dot is one attempt to read the cell back. Now increase the wear: a worn cell holds its charge less precisely.

Each extra bit per cell makes the drive cheaper, but halves the room between levels. Cells also wear out: pushing electrons through the insulation slowly damages it, so each cell survives only a limited number of program and erase cycles. That's why every SSD stores extra bits alongside the data, an error-correcting code that lets the controller detect and fix bits that read back wrong, and why it quietly retires worn-out cells.

The erase problem

Flash has one very awkward rule. The chip reads and writes in pages of a few kilobytes, but a page that has been written can't simply be overwritten. First it must be erased, and erasing only works on a whole erase block of hundreds of pages at once, several megabytes.

Imagine having to wipe an entire shelf every time you wanted to change one word in one book. So an SSD never rewrites a page in place. Instead:

Fill the drive with pages 0–9, then rewrite page 3 a few times and watch where the copies go. Keep writing until garbage collection kicks in, and watch the write amplification.

The extra copying means the flash is written more often than the computer asked for, called write amplification. To keep it down, SSDs hold back spare space the user never sees, and the operating system helps with the TRIM command, telling the drive which blocks belong to deleted files so it doesn't bother copying them. The drive also spreads writes evenly over all its blocks, called wear leveling, so that no block wears out early.

Inside an SSD

All this juggling needs a brain. Every SSD contains a controller: a chip with processor cores of its own, running firmware that manages the mapping table, garbage collection, wear leveling and error correction. Most SSDs also have some DRAM to hold the mapping table. The controller talks to many flash chips at once over parallel channels, which is where the speed comes from.

Fast SSDs connect over PCI Express using NVMe, with the submission and completion queues of Chapter 13. With no head to move and nothing to spin, a random read takes tens of microseconds instead of milliseconds, and a good drive can handle a million of them per second.

Meanwhile, inside a real CPU

To read from an NVMe SSD, the operating system writes a 64-byte command into a submission queue in memory: "read these blocks into this buffer". Then it tells the drive with one store to a memory-mapped "doorbell" register (Chapter 11), here the one for queue 1:

; x86-64: ring the doorbell. rbx = the drive's registers, eax = the new end of the queue
mov     dword [rbx + 0x1008], eax       ; 89 83 08 10 00 00
# RISC-V: a0 = address of the doorbell registers, a1 = the new end of the queue
sw      a1, 8(a0)                       # 23 24 b5 00

The drive fetches the command and copies the data into memory by DMA, then posts a completion entry and raises an interrupt. Everything from Part IV, working together.

The computer this book was written on happens to have one of each kind of drive. Linux marks each as rotational or not:

$ lsblk -d -o NAME,ROTA,SIZE,TRAN,MODEL
NAME   ROTA   SIZE TRAN   MODEL
sda       0 119.2G sata   ADATA SX900               ← SSD
sdb       1   1.8T sata   ST2000DM001-1CH164        ← hard disk, 7,200 RPM

We timed reads of 4 KB from random places on each, one at a time, bypassing every cache. The median times were:

Seagate hard disk (7,200 RPM)     8.8 ms     ≈ 26 million clock ticks at 3 GHz
ADATA SSD (SATA, from 2012)       0.24 ms    about 36 times faster

And that's an old SSD on the older SATA connection. A modern NVMe drive is several times faster still.

Try it yourself

  1. A PC floppy had 40 tracks of 8 sectors of 512 bytes, on one side. How many bytes is that?
    Show answer40 × 8 × 512 = 163,840 bytes, which is exactly 160 KB (160 × 1,024).
  2. At 7,200 RPM, how long does one turn take? How long do you wait, on average, for a sector to come round?
    Show answer7,200 turns per minute is 120 per second, so one turn takes 1/120 s ≈ 8.33 ms. On average the sector is half a turn away: about 4.2 ms.
  3. A hard disk manages 100 random reads of 4 KB per second, or streams 200 MB/s sequentially. How much faster is sequential reading?
    Show answerRandom: 100 × 4 KB = 400 KB/s, or 0.4 MB/s. Sequential is 200 / 0.4 = 500 times faster. That's why keeping data together matters so much on hard disks.
  4. How many charge levels must a TLC cell (3 bits) tell apart? And a QLC cell (4 bits)?
    Show answer23 = 8 levels for TLC, and 24 = 16 for QLC, all squeezed into the same voltage window. Each level gets half as much room as with one bit fewer.
  5. In the SSD demo, the computer only ever changes page 3. Why does the drive still end up erasing blocks?
    Show answerEvery write of page 3 needs a fresh, erased page, and leaves a stale copy behind. Free pages run out after a while, so the drive must collect garbage: copy the valid pages out of some block and erase it, just to make room for more copies of page 3.

What we learned

A disk, magnetic or flash, is just a huge row of numbered blocks. Yet you see folders, file names and documents. In the next chapter we'll build that illusion: file systems.