GPUs and Shaders

At the end of the last chapter, the framebuffer was waiting to be filled. Let's do the arithmetic. A 1920 × 1080 screen has about 2 million pixels. Working out the color of each one, with lighting, textures and shadows, can easily take a few hundred operations. Sixty frames a second makes that tens of billions of operations every second, just for the picture.

Many simple workers

The saving grace is that pixels are independent. The color of one pixel doesn't depend on the color of its neighbor, so they can all be worked out at the same time. Graphics is the perfect job for a crowd.

A CPU is built the other way round: a handful of very powerful cores, each with big caches, clever branch prediction and the ability to juggle instructions out of order, all designed to finish one sequence of instructions as fast as possible. A GPU (graphics processing unit) spends its transistors differently: on thousands of small, simple cores that each run more slowly, but together get vastly more done.

Think of a few brilliant professors against a stadium full of students who can each do one simple sum. For a single hard problem, the professor wins. For a million easy sums, the stadium wins by a mile.

From triangles to pixels

How does a 3D scene become pixels? Everything you see in a 3D game, from a character's face to a mountain, is built from triangles, often millions of them. Each corner of a triangle, a vertex, has a position in 3D space and perhaps a color or other details. The GPU turns them into pixels in a series of steps called the graphics pipeline:

1. Vertex shading: a small program, the vertex shader, runs once for every vertex and works out where it lands on the screen, taking the camera's position and angle into account.
2. Rasterization: special hardware finds every pixel that each triangle covers. Each of those becomes a fragment.
3. Fragment shading: another program, the fragment shader (or pixel shader), runs once for every fragment and decides its color.
4. Output: the color is written into the framebuffer of Chapter 16. A depth buffer remembers how far away each pixel's surface is, so that nearer objects hide farther ones.

The test is wonderfully simple. For each edge, a tiny formula says which side of the line the pixel center is on. If it's on the inside of all three edges, the pixel is covered. Even better, the same three numbers say how close the pixel is to each corner, so they can be used to blend the corners' colors, or anything else attached to them, smoothly across the triangle. And every pixel can be tested independently, perfect for a crowd of simple cores.

Shaders: a little program for every pixel

The steps marked "shader" are programs, written by the game's or app's developers, that the GPU runs at enormous scale. A fragment shader is just a function: it receives the position of one pixel and returns its color. That's all it needs to be, because the GPU will call it millions of times, once per pixel, all in parallel.

Shaders are written in special languages that look a lot like C. Here's a real one, in GLSL, the language of OpenGL and of WebGL in your browser. And since every computer and phone today has a GPU, you can run it right now on yours:

Notice that the program never loops over the pixels. It doesn't know about other pixels at all. It just answers "what color is this one?", and the GPU takes care of asking that question for every pixel at once. Programming this way feels odd at first, but it's the key to the GPU's speed.

Inside a GPU

A GPU's thousands of cores aren't completely independent. They're organized in groups, typically of 32, that all execute the same instruction at the same moment, each on its own pixel. NVIDIA calls such a group a warp; AMD calls it a wavefront. This saves a great deal of hardware: one group needs only one unit to fetch and decode instructions, shared by all 32 lanes.

It has one catch. If the shader contains an if, and some pixels in a group take one branch while others take the other, the group must run both branches, one after the other, with some lanes switched off each time. That's called divergence, and it's why shader programmers avoid complicated branching.

GPUs also deal with slow memory differently. A CPU relies on big caches to avoid waiting (Chapter 9). A GPU keeps many groups ready at once, with enormous register files to hold all their work, and when one group has to wait for memory, it instantly switches to another that's ready to go. The waiting still happens, but it's hidden behind other useful work. Add memory chips that can deliver over a terabyte per second on high-end cards, plus special fixed-function hardware for jobs like rasterization and reading textures, and you have a machine built entirely around doing the same thing to a huge amount of data.

That turned out to be useful far beyond graphics. Since about 2007, with NVIDIA's CUDA, programmers can run general-purpose programs on GPUs. Scientific simulations, video processing and, above all, the training of neural networks are mostly giant piles of independent multiplications, exactly what GPUs are good at. We'll come back to that in Chapter 28.

Meanwhile, inside a real CPU

A GPU doesn't run x86 or RISC-V code. It has its own instruction set, often different for every generation. When a program hands a shader to the graphics driver, the driver compiles it, on the spot, into the GPU's own machine code, just as it did for the shader above.

The CPU's job is to be the manager. It writes a list of commands into memory, "use this shader, these textures, draw these 10,000 triangles", and then rings a doorbell register on the GPU (Chapters 11 and 14). The GPU fetches the commands by DMA and gets to work, while the CPU moves on to preparing the next frame.

CPUs have borrowed a little of the GPU's trick, too. Their vector instructions apply one operation to several numbers at once. Here, eight additions in one instruction on x86, and the RISC-V vector extension, which lets the hardware choose how many numbers to process at a time:

; x86-64 (AVX): add 8 pairs of 32-bit floating-point numbers at once
vaddps  ymm0, ymm1, ymm2        ; c5 f4 58 c2

# RISC-V (V extension): process as many as the hardware can, up to a0
vsetvli t0, a0, e32, m1, ta, ma # 32-bit elements; t0 = how many this time
vfadd.vv v1, v2, v3             # add them all: d7 90 21 02

Eight lanes is still a long way from a GPU's thousands, but the idea is the same: one instruction, many pieces of data.

The computer this book was written on uses the GPU built into its Intel Core i7-12700 processor, a modest one with 32 "execution units". Linux shows its maximum clock speed and its separate engines:

$ cat /sys/class/drm/card1/gt_max_freq_mhz
1500                            ← up to 1.5 GHz
$ ls /sys/class/drm/card1/engine
bcs0  rcs0  vcs0  vcs1  vecs0   ← copy, render, 2 × video codec, video enhancement

Even a small GPU has several engines: one for 3D drawing and computing, one just for copying memory, and dedicated circuits for decoding and encoding video, so watching a film barely touches the shader cores at all.

Try it yourself

1. A 1920 × 1080 screen is redrawn 60 times a second, and each pixel needs 100 operations. How many operations per second is that?
   Show answer

   1920 × 1080 = 2,073,600 pixels. × 60 = about 124 million pixels per second. × 100 = about 12.4 billion operations per second, for a fairly simple scene.
2. In the race, each CPU core is 4 times faster than a GPU lane, but the GPU has 32 times as many lanes. Roughly how much faster is the GPU overall?
   Show answer

   32 times as many workers, each 4 times slower: 32 / 4 = 8 times faster. (The demo shows 7.2 because the last group of GPU lanes has fewer pixels left than lanes.)
3. In the shader playground's gradient, what are the values of uv.x at the left and right edges of the canvas? What color is the top-right corner?
   Show answer

   uv.x goes from 0 at the left to 1 at the right, and uv.y from 0 at the bottom to 1 at the top. At the top right, red = 1, green = 1 and blue = 0.6: a pale yellow.
4. Half the pixels in a group of 32 take the if branch of a shader and half take the else. Why does that slow the group down?
   Show answer

   All 32 lanes of a group execute the same instruction at the same time. So the group runs the if part with half its lanes switched off, then the else part with the other half switched off. It takes as long as both branches together.
5. Why does the rasterizer test each pixel's center rather than the whole square?
   Show answer

   So that every pixel gets a simple yes or no answer, and so that two triangles sharing an edge never both draw the same pixel, or both miss it. A pixel belongs to whichever triangle contains its center. (Special rules settle centers that lie exactly on an edge.)

What we learned

• Pixels are independent, so graphics suits a GPU: thousands of simple cores instead of a few powerful ones.
• 3D scenes are made of triangles; the graphics pipeline transforms their vertices, rasterizes them into fragments, and colors each one.
• Rasterization tests pixel centers against three edge functions, whose values also blend the corners' colors.
• Shaders are small programs that run once per vertex or per pixel, compiled by the driver into the GPU's own machine code.
• GPU cores run in groups (warps) that execute the same instruction; branches cause divergence; memory waits are hidden by switching groups.

That completes Part VI, and with it the whole machine: processor, memory, devices, storage and screen. But when you press the power button, all of it starts out empty and confused. In Part VII we bring it to life, starting with Chapter 18: booting, from the first instruction the processor runs to a loaded operating system.