How AI chips work

Chips stacked like pancakes

An AI chip is only as fast as its memory. So engineers stack memory chips like pancakes and put them right next to the brain chip. In 15 minutes, you learn how, and why it matters.

Brain chip: The chip that does the sums. For AI, it is usually a GPU (graphics processing unit), which does thousands of small sums at the same time.
HBM: High-bandwidth memory. Memory chips stacked into a tower and placed next to the brain chip.
Interposer: A thin slice of silicon under the chips. It works like a small bridge board that joins them.
µm: One micrometer. One millionth of a meter. Chip makers measure tiny gaps with it.

01 · The question

The library across town

A friend works in IT. She says: "Why does an AI chip have memory chips stuck right beside it? Why not use normal memory, like in my laptop?"

You do not know. Picture a student who studies at a desk. If the library is next to the desk, she reads fast. If it is across town, she spends the day on the bus.

AI chips have the same problem. After this lesson, you can explain how engineers solved it.

First, a quick guess.

02 · Your guess

Where does an AI chip keep the data it uses most?

A wrong guess costs nothing. Guessing first helps you remember the answer. Pick one to open the next part of the lesson.

03 · The machine

From a far library to a pancake tower

Press Next step to see each step. Steps 1 to 3 explain the idea. Steps 4 and 5 show how the layers are joined. Step 6 shows real chips.

1The wait

2Stack it

3Place it

4Metal bumps

5Copper to copper

6Used today

04 · Remember

Put the memory next to the brain.

An AI chip reads huge amounts of data all day. If the memory is far away, the chip waits. So engineers stack memory chips into towers and place them beside the brain chip.

Layers in the tallest HBM stack (SK hynix)16

Memory on one Nvidia H200141 GB

Gap with hybrid bonding<1 µm

Library next to your desk

You reach out and the book is there. You read all day. This is HBM beside the brain chip.

Library across town

Every book needs a bus ride. You sit and wait. This is memory far from the brain chip.

Three ideas, one after the other

Stack it. Memory chips are thin. Stack 8, 12 or 16 of them and join them with tiny holes filled with metal. The tower takes little floor space.
Place it beside the brain. The brain chip and the towers sit on one thin silicon bridge. TSMC's version is called CoWoS. The distance is tiny, so data moves fast.
Join it with copper. Today, tiny metal bumps join the layers. The next method, hybrid bonding, fuses copper straight to copper. That puts the connections much closer together.

Is this normal today?

Yes, for top AI chips. Nvidia and AMD both put HBM beside the brain chip. Hybrid bonding is already used inside some AMD chips. For HBM itself, it is still coming. SK hynix says the timing is not sure. Taller stacks also get hot, which is one reason this is hard.

05 · Two checks

Pass two checks to finish

A miss is fine. Each miss gives you a new case.

Check 1 of 2 · Name the idea

06 · Tell a friend

Your one-minute answer

Say it in your own words. Here is a version you can use:

The one-minute version

"An AI chip reads a huge amount of data all day. If its memory is far away, it waits, like a student whose library is across town. So engineers stack memory chips like pancakes. That tower is called HBM. They put the towers right next to the brain chip, on one thin silicon bridge. TSMC's version is called CoWoS. The next step is hybrid bonding. It fuses copper straight to copper, with no bumps between the layers. Nvidia and AMD already use stacked memory in their top AI chips."

HBM	Memory chips stacked in a tower, beside the brain chip.
2.5D	Chips side by side on one silicon bridge. TSMC calls it CoWoS.
Hybrid bonding	Copper fused to copper. No bumps between layers.
SK hynix, Micron, Samsung	Make the memory chips.
Nvidia, AMD	Design the brain chips. They use HBM today.

Try it today. Open the Nvidia H200 page and find the memory line: 141 GB, and 4.8 TB/s (terabytes of data per second). Then ask an AI: "explain HBM like I am 12". Check its numbers.

This lesson explains how the industry works. It is not advice to buy or sell any stock.

Back to all lessons