AI Essentials #4 - Compute, from sand to tokens
When I was using Opus 4.7, every time it didn't do what I wanted, I told myself it was a compute issue (even though I now realize I didn't actually know what compute was). It wasn't until Fable 5 blew through $50 of credits in 30 minutes that I decided I had to understand what compute was and how it works.
Compute is the fuel that AI models need to run their calculations. Training a model or responding to users "burns" compute through tokens, like the way an airplane burns fuel by distance travelled.
AI models work by doing billions of small calculations to predict the next token. A 100-word response could be anywhere from 100 to 300 tokens, depending on the complexity of the words. The number of tokens is the number of rounds of calculations, so 200 tokens for 100 words means 200 rounds of calculations.
Creating that fuel, in the form of compute, requires four main things: chips, servers, buildings and electricity. Since I didn't previously know how chips and servers worked, even at a high level, that's what I'm focusing on here.
Chips
The chips used in AI are generally advanced chips, called GPUs. The process to make a GPU is challenging and made up of four building blocks: crystal manufacturing, design, crystal printing and assembly.
Crystal manufacturing
| Step | Activity |
|---|---|
| A1 | Sand and rock are refined to make a wafer. The rock is melted and purified at extreme heat, over and over, until it becomes a pure liquid. |
| A2 | The liquid is then slowly cooled into a single solid log of crystal, weighing a couple of hundred kilos. The crystal has to be nearly flawless for the switches to work. |
| A3 | The log is cut into thin round slices. Each slice, or disc, is the same size. The disc is polished to a mirror finish to become a wafer. It's a standardized process. |
Design
| Step | Activity |
|---|---|
| B1 | The design is where the chip process stops being standardized. Typically a wafer carries one type of chip, replicated many times across the wafer. |
| B2 | Different chips have different designs, and the design determines how many chips get printed on one wafer. |
| B3 | The design for each chip, replicated many times across the wafer, lays out a mix of cores, memory and some wiring. All three are built from switches. |
| B4 | The cores make the calculations. GPU chip designs can have ~20,000 cores, for doing many simple calculations at once. |
| B5 | The memory holds the numbers the core is working with, both the ones waiting to be used and the results it just produced. This is what the core can't hold because its capacity is being used running calculations (like the exam paper holding your answers as you go through an exam). |
| B6 | The wiring carries electricity between the cores and the memory. |
| B7 | The switches can either conduct electricity or block it, depending on the nudges they get. |
| B8 | Switches flip other switches, and patterns of that flipping are what calculation is. |
Crystal printing
| Step | Activity |
|---|---|
| C1 | Before printing, the wafer is covered with a particular light-sensitive chemical layer. |
| C2 | To print the design, a special machine holds a sheet of glass with a metal coating on top of it. |
| C3 | The metal coating has "gaps" where light can get through, almost like a stencil. |
| C4 | The light that comes through the gaps hits a lens, which shrinks the pattern, and that light "burns" the chemical layer on the wafer. |
| C5 | The chemical layer washes away where the light hits it, leaving exposed wafer. Chemicals flow through those exposed spots, soaking trace amounts of other elements into the crystal. Think of it as water flowing across pavement, seeping in only where there are cracks. |
| C6 | You have many rounds of this "burning" of the chemical layer (with different sheets of glass as templates), washing and soaking of trace elements. The result of this is the switches that flip to make calculations. |
| C7 | Eventually, you have rounds of this process that lay metal on top of the wafer instead. Sometimes that metal is wire and sometimes it's the connection points for the wire on the switches. |
| C8 | Each template sheet of glass used for printing has to have the same atom-level scale. The machine that holds the glass also has to hold it with atom-level precision, otherwise the printing across different templates wouldn't be precise. It would be the equivalent of trying to lay one stamp on top of another but each time missing the edges by a little bit. |
Assembly
| Step | Activity |
|---|---|
| D1 | After printing, the wafer is cut into rectangles. A wafer printed for GPUs will be cut into maybe 60 rectangles. Each rectangle is one chip. |
| D2 | Each chip is tested. To test a chip, a probe card, which is a plate with a bunch of tiny needles on it, is wired into a tester machine. That probe card is lowered, with each needle landing on a metal connection point. On a typical chip, that's hundreds to a few thousand needles. |
| D3 | Electricity is fed in through the needles, pulsing in patterns that flick the chip's switches on and off. |
| D4 | The chip's switches send pulses back out through other needles to the tester machine. The pulses are a combination of 1s (current on) and 0s (current off). If the combination of 1s and 0s doesn't match what the tester machine expects, the chip doesn't pass its test. |
| D5 | If the chip passes the series of tests, it gets stitched together with "towers" of memory chips. The memory chip "towers" hold what doesn't fit on the chip itself, mostly the AI model's own numbers. One way to visualize this is that the GPU is Central Park on the busiest day of the year, and the memory chip towers are the apartment buildings that ring the edges of the park. |
| D6 | The GPU chip and its memory towers are then sealed inside a protective shell. On the outside of the shell, metal connection points are added so that the finished GPU can be mounted onto a board inside a server. |
Having got to this point, you can see that making GPUs is complicated. This complexity is why rapidly increasing the supply of GPUs is challenging, because there are only a few companies in the world that can create them. Once the GPUs are ready, however, they get to work inside servers.
Servers
In AI Essentials #2, I went through the difference between client side and server side. Fortunately, making that decision doesn't require much planning because there are plenty of ways to get server capacity. One of the reasons for that is, unlike chips, servers are relatively easy to make.
At the most basic level, a server is a flat metal box, about the size and shape of two pizza boxes stacked. Inside that box are the parts you need for a computer like you have at work or home, minus the things that humans need but machines don't: for example, a screen, keyboard, speakers and battery.
This flat metal box slides into metal frames called server racks, kind of like a baker sliding trays into a rack. The racks stand in rows inside a building, which then becomes a data center.
A single server has, let's say, 50 laptops' worth of power, which is far more than most jobs need. So it gets divided up and "rented" out. When you have a "job" that needs to be done, what you are renting is a virtual machine.
This renting process is everywhere, but it's not always obvious that is what is going on, since many companies make that process invisible. For example, when someone visits one of my products post launch, Vercel switches on a virtual machine for a few seconds to serve them the page. I never see that.
A funny analogy to this could be: imagine you live in an apartment building where every unit has its own separate entrance. Since you come and go through your own door, there are no shared hallways, no shared spaces. There's nothing you can see from the inside to tell you whether there are ten other units, a hundred units, or none. Each unit is the equivalent of renting out a complete computer on its own. Renting those units by the minute is what cloud is.