Accelerators: GPUs, NPUs and Beyond

Think about everything a modern core does for each instruction (Chapter 25): fetch it, decode it, predict branches, rename registers, track it in the reorder buffer, check it against all the others in flight. The actual addition at the heart of it is a tiny part of the work and of the energy. That's the price of flexibility. An accelerator gives up flexibility for efficiency: it does one kind of job, with almost no overhead, and does it very fast.

We've already met some. The GPU of Chapter 17 began as an accelerator for drawing triangles. Processors have special instructions for encryption, and dedicated circuits that decode video, so a film plays without the cores breaking a sweat. And the newest family of chips, found in data centres, laptops and phones, accelerates the neural networks behind modern AI.

Neural networks are matrix multiplications

An artificial neuron is very simple. It takes a list of input numbers, multiplies each one by a weight, adds the results up, and passes the total through a simple function: for example, "if it's negative, use 0 instead". A layer is many neurons looking at the same inputs, each with its own weights. Write the weights as a grid of numbers, a matrix, and a whole layer is one matrix multiplication. Process many inputs at once and it's a multiplication of two matrices.

A large language model, like the ones behind today's AI assistants, is a deep stack of such layers with billions of weights, called parameters. Producing each piece of text, a word or part of a word called a token, means multiplying through every layer: about two arithmetic operations, one multiply and one add, for every parameter. Training the model in the first place takes vastly more. Almost all of that work is multiply-adds.

Multiplying two N × N matrices takes N³ multiply-adds: a thousand for 10 × 10, a billion for 1,000 × 1,000. But the work is perfectly regular, every multiply-add is independent until the final sums, and every number is used many times. That's a dream job for parallel hardware.

Hardware for multiply-adds

There are several levels of muscle for this one operation:

• SIMD units ("single instruction, multiple data") in ordinary cores. The processor this book was written on has AVX2: one instruction works on 8 numbers at once, and its fused multiply-add instruction does a multiply and an add in one step. RISC-V's vector extension does the same, letting each chip choose how many numbers fit in a vector.
• GPUs, with thousands of simple lanes running the same instructions on different data (Chapter 17). In 2012, a neural network trained on two gaming GPUs beat every other entry in a famous image-recognition contest, and AI has run on GPUs ever since.
• Tensor cores, added to GPUs from 2017: units that multiply small matrices, such as 4 × 4 blocks, in a single operation.
• TPUs and NPUs (tensor or neural processing units): chips, or parts of chips, built only for neural networks. Google's TPU, first used in 2015, is built around a systolic array: a grid of multiply-add cells, each passing its numbers on to its neighbours, so every number fetched from memory is used again and again. Phones have had small NPUs since 2017, and many laptops now have them too, to run AI features without draining the battery.

The processor this book was written on has no separate NPU, but it does have instructions for AI: AVX-VNNI ("vector neural network instructions") multiplies groups of 8-bit numbers and adds them up in one step.

Fewer bits

Programs for science and engineering usually need numbers accurate to many digits. Neural networks don't: they're trained to cope with noise, and a weight of 0.1234 works almost as well as 0.123. So AI hardware uses low-precision numbers. Halving the bits halves the memory needed, doubles how many numbers fit in each memory read, and lets a chip do more multiplications for the same energy. Floating-point numbers, the computer's version of scientific notation, have a sign, an exponent that sets the scale, and a fraction that sets the precision; the formats differ in how many bits they give to each.

Storing a model's weights as 8-bit or even 4-bit integers, each group with a shared scale factor, is called quantisation. It shrinks a model to a quarter or an eighth of its 32-bit size, usually with only a small loss of quality.

The memory wall

Here's the surprise. For running a language model, arithmetic usually isn't the bottleneck. To produce each token, the hardware must read every weight from memory once. A model with 8 billion parameters in 16-bit numbers is 16 GB of weights, and reading 16 GB takes time, however fast the multipliers are. The arithmetic units sit waiting for data: the same problem as Chapter 9's caches, on a giant scale.

That's why AI accelerators come with special memory. HBM (high-bandwidth memory) stacks several memory chips on top of each other and places them right beside the processor, connected by thousands of wires, to deliver several terabytes per second.

Meanwhile, inside a real CPU

How fast can the computer this book was written on multiply matrices? Here's the same job, multiplying two square matrices of 32-bit floating-point numbers, written in different ways. Speeds are in GFLOPS: billions of floating-point operations per second.

Python, plain loops (128 × 128):                  0.066 GFLOPS
C, textbook loop order (1024 × 1024, one core):    0.95  GFLOPS
C, better loop order + AVX2 (1024 × 1024, one core): 26 GFLOPS
the same, on all 20 threads (2048 × 2048):          179  GFLOPS

From the first line to the last, the same arithmetic runs about 2,700 times faster. The biggest single step came from changing the order of the loops. The textbook order walks down the columns of B, jumping 4 KB through memory at every step, which wastes the caches (Chapter 9) and can't use SIMD. Swapping the two inner loops makes every access go along a row, so the compiler can use AVX2 to process 8 numbers per instruction:

for (int i = 0; i < n; i++)          /* textbook: i, j, k */
    for (int j = 0; j < n; j++)
        for (int k = 0; k < n; k++)
            C[i][j] += A[i][k] * B[k][j];    ← B[k][j]: down a column

for (int i = 0; i < n; i++)          /* faster: i, k, j */
    for (int k = 0; k < n; k++)
        for (int j = 0; j < n; j++)
            C[i][j] += A[i][k] * B[k][j];    ← along a row: vectorises

Even so, one core managed only about a sixth of its theoretical peak of roughly 150 GFLOPS: two fused multiply-add units, each handling 8 numbers per cycle. Carefully tuned maths libraries get much closer by splitting the matrices into blocks that fit in the caches. A big data-centre GPU, by comparison, reaches hundreds of thousands of GFLOPS on lower-precision numbers.

The memory side, measured by adding up 1 GiB of numbers:

one core:      7.8 GB/s
8 P-cores:    30.8 GB/s       ← the memory's limit, shared by every core

And a lesson in precision, discovered by accident. That memory test filled the array with 2²⁸ ones, about 268 million, and added them up in a 32-bit float. One core reported a total of exactly 16,777,216:

read 1 GiB in 0.137 s: 7.9 GB/s (sum 1.67772e+07)

16,777,216 is 2²⁴. A 32-bit float has 24 bits of precision, so above that number it can't represent every whole number any more: 16,777,216 + 1 rounds back down to 16,777,216. From then on, every addition of 1 was lost. (With 8 cores, each summed its own share, so the total got stuck at 8 × 2²⁴.) Programs that add up long lists of small numbers have to take care, by adding in a different order or using more bits for the total.

Finally, the AI-related features this processor advertises:

$ grep -o -w -E 'avx2|fma|f16c|avx_vnni' /proc/cpuinfo | sort -u
avx2          ← 256-bit SIMD: 8 floats per instruction
avx_vnni      ← 8-bit dot products for neural networks
f16c          ← converts to and from 16-bit floats
fma           ← fused multiply-add

The whole journey

This is the last chapter, so let's look back at how far we've climbed. We started with a switch that is either on or off (Chapter 1), and built logic gates and an adder from it. Feedback gave us memory (Chapter 3), and a clock and a few registers gave us a toy processor that fetched, decoded and executed its own instructions (Chapters 4 to 7). We gave it memory with addresses, caches to hide how slow that memory is, and virtual memory so every program gets its own world (Chapters 8 to 10).

We connected it to the outside world through ports, interrupts and DMA (Chapters 11 to 13), gave it disks and file systems to remember things (Chapters 14 and 15), and a screen to draw on (Chapters 16 and 17). We watched it boot, and met the kernel that shares it fairly among hundreds of programs (Chapters 18 to 20). We programmed it in assembly, saw how the stack makes functions work, and how compilers, interpreters and garbage collectors let us write in friendlier languages (Chapters 21 to 24). And in this last part, we saw the tricks that make a modern machine fast: pipelines and speculation, many cores, networks, and chips built just for multiplying (Chapters 25 to 28).

Every layer is built only from the layers below it. Nothing in your computer is magic: it's switches, all the way down, arranged with an enormous amount of human ingenuity. Now you know how.

Try it yourself

1. How many multiply-adds does it take to multiply two 1,000 × 1,000 matrices? How long would that take at 179 GFLOPS?
   Show answer

   1,000³ = one billion multiply-adds, which is 2 billion floating-point operations. At 179 billion per second, about 11 milliseconds.
2. A model has 70 billion parameters. How big are its weights in 16-bit numbers? And in 4-bit integers?
   Show answer

   16 bits is 2 bytes: 70 × 2 = 140 GB. 4 bits is half a byte: 35 GB. Quantisation makes the difference between needing several data-centre GPUs and fitting on one.
3. Why does FP16 turn 70,000 into infinity, while BF16, with the same 16 bits, stores it?
   Show answer

   FP16 has only 5 exponent bits, so its largest number is 65,504. BF16 has 8 exponent bits, the same as FP32, so its range is just as huge; it pays for that with only 7 fraction bits, so it stores 70,000 less precisely.
4. An 8-billion-parameter model stored as 8-bit integers runs on a computer whose memory delivers 31 GB/s. About how many tokens per second can it produce, at most?
   Show answer

   The weights are 8 GB, and every token needs them all read once: 31 ÷ 8 ≈ 3.9 tokens per second, whatever the arithmetic units can do.
5. Why did the textbook loop order run 27 times slower than the reordered one, for exactly the same arithmetic?
   Show answer

   Walking down a column of B jumps a whole row's length through memory at every step, so almost every access misses the cache, and the compiler can't use SIMD. Walking along rows uses each cache line fully and lets one AVX2 instruction handle 8 numbers at once.

What we learned

• Accelerators trade flexibility for efficiency: they do one kind of job with almost no overhead.
• Neural networks are mostly matrix multiplications: billions of independent multiply-adds.
• SIMD units, GPUs, tensor cores, TPUs and NPUs do more and more multiply-adds per step.
• Low-precision formats (FP16, BF16, FP8) and quantised integers save memory and energy, at a small cost in accuracy.
• Running large models is usually limited by memory bandwidth, not arithmetic, hence fast memory like HBM.

That's the end of the book, but not of the subject. Every chapter here could be a book of its own, and the machine keeps changing. Go back to any demo and play some more, open up the real tools we used on your own computer, and keep asking the question this book started with: how does that actually work?