Introduction

How to Use This Book

This is a self-study guide compiled for the semiconductor sessions of Sunday AI, a study group, and covers AI infrastructure and semiconductors in seven sessions. It is written for readers with a background in statistics, computing or AI who never studied semiconductors. Instead of asking you to memorize terms, it sets up a handful of equations and has you plug in the numbers yourself. Every session comes with calculation exercises and full solutions, so you can check your answers on your own. In the screen edition (HTML), interactive figures placed next to the equations let you change the values; the print edition (PDF) carries the same figures in their initial state at the same spots. You are welcome to read it the same way in another group, or alone.

What this book covers

The seven sessions follow one question: "Where are the limits of the physical equipment that runs AI set?"

The two equations set up in Sessions 2 and 4 govern everything else. If you are short of time, work through at least those two by hand.

What each session contains

Every session follows the same structure.

The solutions in appendix A are folded. On screen, click the heading to open them; when you print, they all open automatically.

An eight-week schedule

There are seven sessions, but eight weeks works better. Session 5 (front-end processes) is one and a half times as long as the others.

That is about two hours a week. If one session is too much at a sitting, break it at the steps in that session's "Reading order" table.

Update the same picture every week

What breaks down most easily in self-study is "I read it, but nothing stayed." There is one remedy. Draw the skeleton below on one sheet of paper (or one screen), and add the layer you learned each week on top of it.

[request] → [model/software] → [compute chip] → [memory] → [package]
                                                                   ↓
[power/cooling] ← [DC/rack] ← [server/board] ← [fab/supply chain] ←┘

Three rules for updating it

1. Put one number on every arrow. (For example, on the arrow from compute chip to memory: "ridge point ≈ 295 FLOP/byte".)
2. Mark on every box whether it is the bottleneck. As the sessions go on, you should see the bottleneck mark move.
3. Do not erase last week's picture. Draw over it in a different color.

"The seven sessions in one line" at the end of Session 7 is the final answer key. If in week 8 you can reconstruct it without looking, you are done.

When you get stuck

A note on the numbers (read this)

The calculation exercises in this book use many assumed values. They are examples meant to show the direction of a conclusion, not measurements.

The reference date is October 8, 2026. Statements that change over time, such as volume-production and shipment status, power-demand projections, standards and product specifications, were rechecked on that date against primary sources (manufacturer announcements, standards documents and agency reports), and the text gives the date each was checked.

Assumed values that must not be quoted as facts

• The price of one wafer (USD 15,000), electricity (USD 0.08 per kWh), the price of an accelerator node (USD 250,000)
• ASIC and GPU unit prices and NRE, defect density $D_0$, assembly yield, interface overhead ratios
• Any other settings stated in individual exercises

Values taken from public spec sheets that change with each generation

• Compute performance and memory bandwidth of accelerators (A100, H100, B200). All are dense (no sparsity) figures. Vendor material also lists a figure twice as high with sparsity, so always check which one you are looking at
• HBM specifications, the reticle limit, microbump and hybrid-bonding pitch

Things that are structurally stable and safe to quote

• The equations themselves: the roofline, the yield model, the Rayleigh criterion, thermal resistance, the definition of PUE
• The direction of the relationships that follow from them: yield falls exponentially as die area grows, depth of focus worsens with the square of NA, small batches are bandwidth-bound, and so on

This book deliberately leaves out absolute figures for market size and market share. Each source defines and counts them differently, so they vary widely. Section 4 of Session 1 explains why.

Read alongside other material

The thinner parts of this book are the following.

Session 1

Mapping the AI Semiconductor Industry

Learning objectives

By the end of this session you should be able to answer the following.

1. When you see a news story saying "the AI chip market is worth trillions of dollars," what should you check first?
2. How do profit margins and irreplaceability differ across the layers of the value chain?
3. As demand moves from training to inference, how do the hardware requirements change?
4. Into what types do semiconductor supply-chain risks fall, and how do they differ in character?

0. Self-study guide — for readers working alone

Prerequisite check — can you explain each of these five in one line?

If any of them stops you, read the answer below and move on. That is all you need for this session.

Reading order (100 minutes recommended)

When you read alone, front to back is not the best order.

Never skip: "Why this shift put memory on the throne" in section 3. The single line "arithmetic intensity $I\approx B$" found there leads straight to the roofline in Session 2, HBM in Session 3, packaging in Session 6, and the power problem in Session 7. If this paragraph does not make sense, Session 2 will keep sliding past you.

If section 3 is hard — a detour

Section 3 borrows ideas that have not yet been set up (arithmetic intensity, ridge point), so getting stuck on a first reading is normal. Get a feel for it with the analogy below, move on, and come back after reading Session 2. It will open up.

Suppose you fetch an entire book from the library, read one sentence, and send it back. Moving the book takes ten minutes; reading takes one second. The bottleneck is not "reading speed" but "moving speed." Decoding during inference is exactly like this. To produce one token, the whole set of model weights is read from memory. However fast the chip is, it makes no difference; how fast you can haul the weights in from memory decides everything. There is only one fix: once the book has been brought over, have many people share the reading (batching). This is where the importance of batching comes from.

1. Getting the intuition — what "AI semiconductor" actually means

It is not one thing

The term "AI semiconductor" refers to at least four things.

When an article or report says "the AI chip market is worth X billion dollars," it often does not say which of the four it means. Different definitions produce numbers that differ several-fold. The first habit this book tries to build is asking "which scope is this?" every time you see a number.

Why this industry suddenly grew so large — three lines

1. A transformer is a lump of matrix multiplications. Its computation reduces to massively parallel multiply-accumulate operations, so the gain from dedicated hardware is extreme.
2. Scaling laws gave empirical support to "bigger model = better performance." The belief that compute input translates into performance justified the capital spending.
3. But only a handful of places can make that hardware. Demand exploded while supply had no physical or financial elasticity. So prices and profits piled up at particular points.

You can read this book as a seven-session dissection of the "why only a handful" in point 3.

2. The value chain — who makes money where

Three layers

Between layers 1 and 2 of the figure sits one more thin layer. Design houses (design service companies) take the logical design handed over by a fabless company, implement and verify it physically for a particular foundry's process, and pass it on to manufacturing. Foundries tie them into partner networks. TSMC describes the members of its VCA (Value Chain Alliance; formerly Value Chain Aggregator) as "independent design service companies working closely with TSMC to help system companies, ASIC companies, and emerging start-up customers bring their innovation to production." They "provide specific services at each link in the IC value chain, including IP development, design backend, wafer manufacturing, assembly, and testing" (checked October 2026). The players and each foundry's programs are covered in Session 4.

Key insight — profit margin and irreplaceability are different things

• Highest margins: the leading fabless companies in layer 1 and the leading foundry in layer 2
• Highest irreplaceability: layer 3. EUV lithography tools in particular come effectively from ASML alone. Its revenue is far smaller than NVIDIA's, but if it stops, all of leading-edge logic stops.

This asymmetry explains why geopolitical controls do not stop at finished chips but also aim at layer 3 (equipment). Controls on finished chips block the flow of chips sold today; controls on equipment tie up the ability to make such chips yourself (section 5 ③).

Where Korea stands

This book was written for a study group in Korea, so the original keeps a short note on the country's position. It is kept here because it shows how the three-layer picture reads from one country's vantage point.

• Strong: memory (DRAM, NAND, HBM). SK hynix and Samsung Electronics hold a large share of the world market. For example, by revenue in the first quarter of 2026, SK hynix ranked first in the world in HBM and second in DRAM (including HBM) and in NAND (IDC figures, as given in the prospectus SK hynix filed for its U.S. listing in July 2026; checked 2026-10-08).
• Relatively weak: share of leading-edge foundry, the fabless ecosystem, the domestic share of equipment and materials.
• Structural implication: memory has traditionally been a market with violent cycles. Whether AI demand has changed that cycle structurally or only widened its swings is the final discussion topic of Session 7.

Comparable cases elsewhere. Taiwan reads the same picture from layer 2. TSMC reported that it "represented 34 percent of the Foundry 2.0 industry, which we define as all logic wafer manufacturing, packaging, testing, mask-making and others, output value in 2024, as compared to 28 percent in the previous year" (TSMC 2024 business report, in the minutes of its June 3, 2025 shareholders' meeting). The definition is the company's own, so the share is also a small case of the problem in section 4: change the denominator and the number changes. (source)

3. The shift in demand — from training to inference

The concern behind every later session comes from this section.

Training and inference want different machines

Why this shift put memory on the throne

Training uses large batches, so it can keep the compute units busy. But in the decoding phase of inference, producing one token requires reading the whole set of model weights from memory. Each read of the weights supports only as many operations as there are sequences in the batch.

Roughly speaking, the amount of computation per byte of fp16 weights is proportional to the batch size $B$. Arithmetic intensity is about $I\approx B$. As Session 2 will show, the ridge point of current GPUs is around 300 FLOP/byte. With a batch below 300, the GPU is bound by bandwidth and its compute units sit idle.

Yet latency requirements keep the batch from growing without limit, and the KV cache eats memory, so batch size is capped by memory capacity. In the end, the economics of an inference service are decided not by "how fast the chip is" but by "how wide and how large a memory is attached to it."

This is why HBM became the center of the industry (Session 3), why advanced packaging became a bottleneck (Session 6), and why data center power became a problem (Session 7).

4. How to read market figures critically

What actually happens

For the "AI accelerator market" of the same year, different sources give very different figures. The reasons:

1. Differences in scope — Is it the accelerator chip only, the module and board too, or the whole AI server? Counting the whole server brings in the CPU, memory, network and power supply, and the number grows several-fold.

   Power figures hide the same trap. NVIDIA lists the maximum thermal design power (TDP) of the H100 (SXM) as "Up to 700W (configurable)", and gives the maximum input power of the DGX H100 server, which holds eight H100s, as 10.2 kW. Do the arithmetic: the TDP of eight chips adds up to 5.6 kW, and the gap to the server's maximum input power is 4.6 kW. Items such as the CPUs, memory, network, fans and power-conversion losses fall into that gap. Both values are ceilings, however, so this does not mean that 4.6 kW is used outside the chips during actual work. This session does not break the gap down into numbers. The next-generation DGX B200 has a maximum system power of about 14.3 kW. Each time you widen the boundary, from chip to server and from server to a facility that includes cooling and power distribution, the number grows. When you see the phrase "the power one AI chip uses," first ask at which boundary it was measured. The question comes back in the power-cost calculation of Exercise 4 in Session 2. Manufacturing has the same structure: wafer capacity, number of good dies, package output and shipments are different numbers.

2. Differences in the unit counted — Revenue, units shipped, or deployed compute (FLOPS)? In-house chips (such as TPUs), for example, are not sold outside, so they are underestimated in revenue-based statistics and come out much larger when counted in deployed FLOPS.

3. The interests of whoever publishes — chip companies, market research firms and investment banks each have different incentives.

4. Differences in timing — this is a market whose forecasts are revised heavily every half year.

What can and cannot be pinned down

This book draws the line as follows.

Structurally settled (several sources agree)

• NVIDIA holds the overwhelming majority of data center AI accelerator revenue. What the ratio is and which way it is moving depends on how you set the denominator, so it is not given here (Exercise 1).
• The share of hyperscalers' in-house ASICs (Google TPU, AWS Trainium, Meta MTIA, Microsoft Maia) is rising fast, and Broadcom, Marvell and others have risen as their design partners.
• Inference has overtaken, or is overtaking, training as a share of AI compute spending.
• HBM is an oligopoly of three companies (SK hynix, Samsung, Micron), and SK hynix leads by revenue (confirmed with IDC figures for the first quarter of 2026, 2026-10-08). The order and gap between second and third place swing with each product generation, so they are not stated here.

Not settled (sources vary widely — mention only as ranges)

• Absolute market size by year
• Exact percentage shares by company (for example, the same company's HBM share differs by several percentage points depending on whether it is a quarterly result or an annual forecast, and whether it is counted by revenue or by shipments)
• Total data center power consumption. Even authoritative estimates span a wide range. Lawrence Berkeley National Laboratory (LBNL) estimated U.S. data center electricity use in 2028 at 325–580 TWh (section 6, misconception 6)

If you read this in a group: make it a rule that the presenter must state "the source and definition of this number in one line." The learning effect of this session rises sharply.

5. Three axes of supply-chain risk

① Geographic concentration

Leading-edge logic manufacturing is heavily concentrated in Taiwan. That makes it a single point of failure, both for natural disasters and for geopolitics. Countries' policies to attract fabs (the U.S. CHIPS Act, the EU Chips Act, support in Japan and Korea) try to ease this concentration, but they are not short-term fixes: building one fab and raising its yield takes years.

② Single-supplier bottlenecks

The important point here is that "can it be made technically" and "how much can actually be made" are different questions. For instance, the supply of AI accelerators runs into limits on CoWoS packaging capacity and HBM supply before it runs into limits on logic wafers. (Covered in detail in Session 6.)

Bottlenecks also form a chain on the demand side. EUV → CoWoS → HBM is the movement of the bottleneck seen from the supply side. The same movement can be traced from the demand side. As models grow and contexts lengthen, both compute and the KV cache grow, and so do the capacity and bandwidth demanded of the HBM that holds the weights and the KV cache. Attaching more HBM makes the package larger and more complex. Pack such accelerators densely into servers and the heat density rises, until the power and cooling of the racks and the facility reach their limits. When demand at one link rises, the pressure passes to the next. So what constrains the system is usually not performance within one layer but the interfaces where layers meet: between memory and chip, chip and package, server and facility. Session 3 (HBM), Session 6 (packaging) and Session 7 (servers and facilities) each take one of these links.

③ Policy risk

Export controls target equipment and manufacturing capability as well as finished chips. Because of extraterritorial reach (the foreign direct product rules and others), firms in third countries are affected too. As of October 2026, the text of the foreign direct product (FDP) rules in the U.S. Export Administration Regulations (EAR) (15 CFR §734.9) contains a separate advanced computing FDP rule (paragraph (h); advanced integrated circuits and computers) and a semiconductor manufacturing equipment (SME) FDP rule (paragraph (k)). The structure is that an item made with certain U.S. technology or software, or made in a plant built with such technology, can be subject to the EAR even if produced outside the United States. The text can be read through the eCFR link under "Further study" in section 8.

"The paradox of the supercycle"

The stronger the boom, the more everyone invests in expanding the bottleneck process. Because fabs and equipment have lead times of two to three years, the demand signal at the moment of the investment decision and the demand at the moment the capacity comes online diverge. This is the semiconductor industry's classic bullwhip effect, and it is especially severe in memory. One camp argues that AI demand has abolished this cycle; the other, that it has only widened the swings. (Final discussion of Session 7.)

6. Common misconceptions

Misconception 1: "AI chip = GPU" → GPUs dominate today, but the share of ASICs such as TPU and Trainium is rising fast. More important, memory (HBM) and packaging account for a large part of the performance and cost of an AI system.

Misconception 2: "The smaller the nanometer number, the better the chip, always" → Node names are marketing labels, not physical dimensions (Session 4). AI accelerator performance also depends more on memory bandwidth and packaging than on the node.

Misconception 3: "With enough money, you just build a fab" → Equipment lead times, skilled labor and the yield ramp (the learning curve to reach volume-production yield) are bigger constraints than capital. A fab building goes up in two years, but bringing yield to a competitive level takes longer.

Misconception 4: "China will soon catch up / will never catch up" → Both are oversimplifications. China has already secured considerable capacity at mature (legacy) nodes, while at the leading edge of logic it faces the structural constraint of having no EUV. It has to be looked at layer by layer.

Misconception 5: "The AI chip boom will end soon / will last forever" → The honest answer is: "Because the nature of demand has shifted from training to inference, it is uncertain whether the old cycle models still apply." Inference demand is more continuous and more like an operating expense than training, which could dampen the cycle, but the structure of capacity lead times is unchanged.

Misconception 6: "Once power-efficient chips arrive, total data center power will fall" → The efficiency of one chip and the total for a whole facility are different questions. According to LBNL's 2024 report on U.S. data center energy use, prepared with support from the U.S. Department of Energy, U.S. data center electricity use grew from 58 TWh in 2014 to 176 TWh in 2023, about 4.4% of total U.S. electricity in 2023. For 2028 it is estimated at 325–580 TWh, 6.7–12% of U.S. electricity (the share depends on how fast the rest of the economy grows). Use more than doubled between 2017 and 2023 alone, and the report names the growth of AI servers as the main cause. Over the same period, chips became more power-efficient with each generation. Yet the total tripled, so efficiency and total can move separately (this interpretation is this book's own, not a conclusion taken from the report). Note also that the 2028 estimate spans nearly a factor of two. That is why section 4 classes the total as "not settled" and gives it only as a range.

7. Discussion questions

1. From a country's point of view, in which of the three layers of the value chain is it most efficient to put investment and policy resources? How does the answer change with the time horizon (5 years vs. 20 years)?
2. Is it rational for hyperscalers to build their own chips? Under what conditions is it rational, and under what conditions not? (Taken up quantitatively in Session 4.)
3. Are the wide differences among estimates of "the AI chip market size" a problem of measurement or a problem of definition? As someone trained in statistics, how should you treat these estimates?
4. How should we understand the bottleneck in the supply chain moving from EUV (equipment) to CoWoS (packaging) and then to HBM (memory)? Why does the bottleneck keep moving?
5. Is there evidence that the traditional memory cycle (boom → expansion → glut → slump) has been changed structurally by AI demand?

8. Further study

• Overview of the industry structure: Chris Miller, Chip War (2022) — the standard popular book on the geopolitics of the semiconductor industry. Good background reading for this session
• Regular tracking of technology trends: industry analysis from SemiAnalysis, TrendForce, Counterpoint Research and others. Allow for each one's slant
• Primary material: quarterly earnings materials from TSMC, SK hynix, Micron and NVIDIA (especially the Q&A of the earnings calls). Often more accurate than market reports
• Korean perspectives (in Korean): semiconductor industry analysis from the KDI Economic Information and Education Center, the Korea Institute for Industrial Economics and Trade (KIET), and ETRI's Electronics and Telecommunications Trends
• Power demand: LBNL press release on the 2024 U.S. data center energy report — source of the figures in section 6, misconception 6
• Export-control text: eCFR 15 CFR §734.9 — current text of the foreign direct product (FDP) rules

Self-study appendix A — number-audit exercises and solutions

This session has no calculation exercises. Instead it has three problems that put the "ask for the definition first" habit from section 4 to work. Write your answer on paper before you look.

Exercise 1 — Same company, different numbers

One article says "Company A's share of the AI accelerator market is 90%," and another says "70%." Assuming neither article is lying, what differences in definition could produce this gap? Give at least three.

Exercise 2 — The growth-rate trap

In the same year, one article writes "the AI chip market grew 200% year on year," and another writes "the semiconductor market grew 25% year on year." Both are about the semiconductor industry. Is this a contradiction? Before answering, write down what each sentence counted.

Exercise 3 — Locating the bottleneck

Suppose the following three observations are reported at the same time.

• Leading-edge logic wafer capacity has room to spare.
• Lead times for accelerators exceed 40 weeks.
• HBM is pre-contracted through next year's volume.

Where is the bottleneck? And why would a policy proposal to "expand wafer capacity" have no effect?

Self-study appendix B — leads for the discussion questions

The discussion questions in section 7 have no right answers. But a reader working alone may not even know what to base an answer on, so only the axes of the argument are given. The conclusions are yours to draw.

Self-study appendix C — key terms

Only the terms that first appear in this session. They are used from Session 2 on.

Self-study appendix D — self-assessment

If you can answer each item in your own words, you are ready for the next session. If two or more stop you, go back to the relevant section in the "Reading order" above.

• ☐ Given an article on "the AI chip market size," I can immediately name three things to check → sections 1 and 4
• ☐ I can draw the three layers of the value chain and explain why margins and irreplaceability diverge in each → section 2
• ☐ I can say in one sentence why export controls aim at manufacturing equipment as well as finished chips, and what equipment controls tie up that chip controls do not → sections 2 and 5
• ☐ Without looking at the table, I can name four or more ways in which the hardware that training and inference need differs → section 3
• ☐ I can explain why "reading the whole set of weights to produce one token" makes memory the bottleneck → section 3 ★
• ☐ I can say why a larger batch raises efficiency and why it cannot grow without limit → section 3 ★
• ☐ I can tell the three axes of supply-chain risk apart and explain why the remedy for each is different → section 5
• ☐ I can explain the movement of the bottleneck from EUV → CoWoS → HBM as "normal operation" → section 5

The two ★ items must be passed. All of Session 2 is a refinement of those two sentences.

Before the next session

Three things were named in this session but left undefined. Session 2 sets them up properly.

The question that carries over: if memory that cannot keep up makes a faster chip useless, where exactly is that limit drawn?

Sources

Introductory material chosen by the study curriculum (the "AI semiconductors" item)

Primary sources added in edition 2.0 (checked 2026-10-08)

• TSMC, Value Chain Alliance — design houses in section 2
• SK hynix, prospectus for its U.S. listing (Form 424B4, 2026) — original on SEC EDGAR. By revenue in Q1 2026: first in HBM (56.4% share), second in DRAM including HBM (29.1%), second in NAND (18.5%). All cite IDC figures (checked against the original 2026-10-08)
• NVIDIA H100 product specifications · NVIDIA DGX H100/H200 User Guide, Table 3, power specifications · NVIDIA DGX B200 specifications — measurement boundaries in section 4
• LBNL, "Berkeley Lab Report Evaluates Increase in Electricity Demand from Data Centers" (2025-01-15) — section 6, misconception 6
• eCFR, 15 CFR §734.9 Foreign-Direct Product (FDP) Rules — section 5 ③

Verification status

• The three-layer value chain and the main firms in each layer — standard industry classification. Verified
• The nature of design houses and TSMC's VCA — confirmed on TSMC's official page (2026-10-08; quoted wording rechecked 2026-10-22). The earlier name "Value Chain Aggregator" appears as the page title in Internet Archive copies of 2019-07-30 and 2020-11-16, and "Value Chain Alliance" from the 2022-09-25 copy on (checked 2026-10-22). The member list changes over time and is not given in the text
• ASML as the sole supplier of EUV lithography tools — verified (several sources agree)
• The three-company HBM oligopoly and its leader (SK hynix) — reconfirmed with IDC figures for Q1 2026 by revenue (2026-10-08, SK hynix prospectus). The order of second and third was not confirmed from a primary source and is not given in the text. Exact share figures vary widely across sources and are not asserted in the text
• H100 SXM TDP "Up to 700W (configurable)", DGX H100 maximum input power 10.2 kW, DGX B200 maximum system power about 14.3 kW — checked against the manufacturers' original specifications (2026-10-08; the H100 wording rechecked 2026-10-22). 5.6 kW and 4.6 kW are calculated values
• U.S. data center electricity 58 TWh (2014), 176 TWh (2023, about 4.4%), 325–580 TWh (2028 estimate, 6.7–12%), more than doubling 2017–2023 and its main cause (AI servers) — checked against the LBNL press release (2026-10-08)
• The advanced computing FDP rule and the SME FDP rule in 15 CFR §734.9 — current eCFR text checked (version updated as of 2026-10-05, read 2026-10-08). That the items in paragraph (h) are advanced integrated circuits and computers such as ECCN 3A090 and 4A090, and that paragraph (k) covers manufacturing equipment such as 3B001, was rechecked against the 2026-10-01 version of the text through the eCFR API (2026-10-08)
• The shift in which inference accounts for most AI compute spending — consistent across several industry analyses, but the exact timing and ratio vary by source. Described only as a structural trend
• Absolute size of the AI accelerator market — unconfirmed, wide variation. No specific figure is given in the text
• Exact percentage shares by company — unconfirmed, wide variation. Only the structural ranking is described

Session 2

AI Chips — Compute Semiconductors

Learning objectives

1. Explain why CPUs are inefficient for AI computation in terms of energy, not "because the clock is slow."
2. Tell apart how GPU tensor cores and TPU systolic arrays each gain their efficiency.
3. Build the roofline model and determine by calculation whether a given workload is compute-bound or memory-bound.
4. Weigh the break-even point of developing an in-house ASIC quantitatively.

0. Self-study guide — for readers working alone

What to bring from the previous session

Prerequisite check

Reading order (120 minutes recommended)

If section 6 is hard — a detour in three sentences

The roofline has a grand name, but its content is grade-school arithmetic.

1. To get work done you have to bring the materials in (bandwidth $B$) and process them (compute $P_{\text{peak}}$).
2. Whichever of the two is slower sets the overall speed. Hence the $min$.
3. Which one is slower depends on "how long you process 1 kg of material" ($I$). If processing takes long, compute is the bottleneck; if it is over quickly, transport is.

$$P_{\text{attainable}}(I)=min(P_{\text{peak}},B\times I)$$

That is all the equation says. The rest is plugging in real numbers.

1. Getting the intuition — computing is cheap, delivering data is expensive

A logistics analogy

Suppose machining one part in a factory takes 1 second. If fetching that part from the warehouse takes 3 minutes, adding more production lines will not raise output at all. There are only two ways to improve.

• Move the warehouse closer (= on-chip SRAM, HBM placed right next to the chip)
• Do several jobs with each part you fetch (= data reuse, tiling, larger batches)

Almost every idea in AI accelerator design is one of these two. Everything else is detail.

The shock in numbers

According to Horowitz's classic summary (ISSCC 2014, for a 45nm process), the energy gap between computation and memory access looks like this.

The top three rows (add and multiply) are taken directly from Figure 1.1.9 of Horowitz's paper. The two memory rows are the 32-bit access values of Han et al. (2015), who cite the same 45nm table. Horowitz's figure gives memory for 64-bit access: cache 10 pJ at 8 KB, 20 pJ at 32 KB and 100 pJ at 1 MB, and DRAM 1.3–2.6 nJ. The SRAM value of 5 pJ is half the 8 KB cache value, so it grows as the cache grows.

So reading data once from DRAM costs about 170 times as much as one floating-point multiplication. Halve Horowitz's DRAM range to get 32-bit values, and the ratio is about 180–350 times. As processes shrank, the energy of computation kept falling, but the energy of off-chip memory access fell much less, so the gap has widened over time.

The conclusion that follows: the objective function a chip designer must optimize is not FLOPS but "total energy per useful operation," and its dominant term is data movement.

2. Why the CPU will not do

First, clear away a misconception

"CPUs are slow because they have few cores" is only half right. The more fundamental reason is the fixed cost the CPU pays per instruction.

What actually happens when a CPU executes the single line c = a * b:

1. Instruction fetch (I-cache access)
2. Instruction decode
3. Branch prediction, register renaming, out-of-order scheduling
4. Register file read
5. The actual multiplication ← only this is "work"
6. Write-back of the result, commit

Steps 1–4 and 6 use far more energy and area than step 5. The CPU is designed on the assumption that "it does not know what comes next," and the cost of preparing for that uncertainty is the overhead.

But deep learning does know what comes next

Matrix multiplication has no data-dependent branches, its access pattern is fixed statically, and the same operation repeats billions of times. With no uncertainty, the whole cost of preparing for it can be removed. That is the entire reason accelerators exist.

Steps 1–6 above describe a single scalar instruction. Real CPUs also load several operations onto one instruction. A SIMD instruction processes a whole vector, and Intel AMX goes all the way to matrices. According to Intel's AMX code-sample article, AMX consists of tiles, which are two-dimensional registers, and TMUL, an accelerator that multiplies tiles as matrices. In the palette 1 configuration there are 8 tile registers totaling 8 KB, and one tile is at most 16 rows × 64 bytes. The same article names 4th Gen Xeon Scalable processors and the Xeon Max series as the processors it applies to. So the description of a CPU handling operations one at a time is an introductory shorthand. The conclusion does not change, though. The CPU still carries the control machinery it needs for generality, so it cannot amortize the same work as deeply as an accelerator.

In short: going from CPU to GPU to ASIC, you "sell flexibility to buy efficiency." The terms of that trade are the ASIC economics of section 5.

3. What a GPU really is — what one line of matmul does in hardware

The SIMT execution model

A GPU runs threads in bundles of 32 (a warp), and the 32 threads of a warp execute the same instruction at the same time. The cost of fetching and decoding an instruction is spread over 32 operations, so the overhead becomes 1/32.

Readers who have written GPU code will know the performance traps that come from this.

• Warp divergence: if threads in the same warp take different branches, the two paths run one after the other, and threads that do not belong to the path being run sit idle. If the threads split half and half between two paths of equal length, throughput halves; the size of the loss depends on the path lengths and the split ratio. → This is what the advice "avoid if on the GPU" really means.
• Memory coalescing: global memory is read in 32-byte transactions. If the 32 threads of a warp each read one consecutive 4-byte value, that is 128 bytes in all and takes four transactions; if the addresses are scattered 32 bytes or more apart, it takes one per thread, 32 in all. → This is why tensor layout (NCHW vs NHWC) decides performance.
• Occupancy: the GPU hides memory waits by running other warps, so the number of warps that can be resident at once matters. GPUs have L1 and L2 caches and shared memory too, but caches smaller than a CPU's cannot remove all of the latency, so much of the latency is hidden by parallelism.

Tensor cores — amortization taken to the limit

A tensor core performs the multiply-accumulate of small matrix blocks, $D=A\times B+C$, as one instruction. For a $16\times 16$ block product, say, one instruction covers thousands of MACs, so the instruction overhead effectively goes to 0.

One line of torch.matmul, seen as a hierarchy:

torch.matmul(A, B)
  └─ cuBLAS/cuBLASLt kernel selection (tile size, split strategy)
       └─ grid/block decomposition: split the output matrix into tiles
            └─ load tiles from global memory (HBM) → shared memory (SRAM)
                 └─ shared memory → register fragments
                      └─ tensor core MMA instructions (block multiply-accumulate)
                           └─ accumulated results passed back up

The key point: this whole hierarchy exists for one purpose, "reuse each tile fetched from HBM as many times as possible." A block product with tile size $T$ reads $O(T^2)$ bytes and performs $O(T^3)$ operations, so the arithmetic intensity grows as $O(T)$. Tiling is the software way of raising the roofline's $I$.

The precision ladder

AI accelerators gain throughput roughly by multiples as precision drops (FP32 → TF32 → BF16/FP16 → FP8 → FP4). There are two reasons.

1. Arithmetic-unit area: the area of a multiplier is roughly proportional to the square of its bit width, so halving the bits lets you fit about 4 times as many in the same area.
2. Memory traffic: half the data means half the bandwidth demand. The real gain from quantization comes from bandwidth, not computation. This is why quantization matters so much in inference.

4. TPUs and systolic arrays — building reuse into the hardware

Structure

A systolic array lays out processing elements (PEs) in a grid and streams data through it, with each PE passing values on to its neighbor. The name comes from the way data flows in a regular rhythm, like blood pushed out by the heart (systole).

In an $N\times N$ array, one input element is reused $N$ times as it passes through the array. That is, each piece of data read once from memory is used for $O(N)$ operations. The arithmetic intensity is guaranteed by the wiring, not by software.

Google's TPU v1 arranged $256\times 256$ = 65,536 8-bit MAC units this way. Data moving inside the array travels over short wires between neighboring PEs, so the energy is extremely low.

The figure below shows the fourth beat when one token ($M=1$, decoding) is streamed through a $4\times 4$ array whose weights sit in the PEs. In the screen edition you can step through the beats one at a time, or change the number of tokens $M$ and the matrix size, and count how many PEs are working at each beat.

GPU vs. systolic array — what was given up and what was gained

The weakness of the systolic array: if the matrix size is not a multiple of the array dimension, the PEs at the edges sit idle. When the batch is small or the matrix is long and thin (decoding!), utilization falls a long way. This is why designs dedicated to inference take a different form from those for training.

Counted in beats, it looks like this. With the weights fixed in the PEs, streaming $M$ tokens through one $K\times N$ tile takes $M+K+N-2$ beats to fill and drain. At TPU v1 size ($K=N=256$), with $M=1$ the PEs are working only 0.2% of the time on average over 511 beats; with $M=\mathrm{4,096}$ that rises to about 89%. Real chips feed the next tile in right behind the current one to overlap the fill and drain time, and loading the weights into the PEs takes time of its own. Still, when $M$ is small each weight is used only $M$ times and then discarded, so overlapping gains little. There are also several dataflows besides weight-stationary, such as output-stationary (the textbook by Sze et al. in section 11).

5. NPUs and ASICs — the economics of specialization

NPU

A neural-network accelerator for mobile and edge devices. Its objective function is different.

• What it optimizes is not absolute performance but TOPS/W (performance per watt)
• Centered on low precision (INT8/INT4), under battery and heat constraints
• Integrated in an SoC with the CPU, GPU and ISP, sharing memory (LPDDR)
• Use is intermittent, so latency and wake-up overhead matter

The fundamental difference from server accelerators: a server spends power to get throughput, whereas at the edge the power budget is fixed and you squeeze out the most performance inside it. Both are "AI chips," but their design goals are close to opposite.

Calculating the ASIC break-even point (exercise)

Whether to build a dedicated chip or buy one can be quantified.

• $C_{\text{NRE}}$ : non-recurring engineering cost (design staff, EDA licenses, IP licenses, mask set, verification and prototypes). At leading-edge nodes the mask set alone runs to tens of millions of dollars, and total NRE can reach hundreds of millions of dollars.
• $c_A$ : ASIC unit cost (wafer cost ÷ number of good dies + packaging + test)
• $c_G$ : unit acquisition cost of the alternative (a commercial GPU)
• $N$ : expected volume

Here we assume that one ASIC does the work of one GPU at the same throughput, latency and quality (1:1 replacement). Only then does the same volume $N$ mean the same demand.

Comparing total costs: $C_{\text{NRE}}+N{c}_A<N{c}_G$

$$N^{*}=\frac{C_{\text{NRE}}}{c_G-c_A}$$

If the ASIC's throughput is $r$ times the GPU's, the same demand needs $N/r$ ASICs. In that case put $c_A/r$ in place of $c_A$ (if $r<1$, the break-even volume grows).

But this formula is only half right. In practice three more terms have to go in.

1. The time axis: developing a chip takes 2–3 years, and model architectures change faster than that. When the chip is finished, will that workload still dominate?
2. Electricity: in a data center, several years of electricity bills can exceed a chip's acquisition price in its total cost of ownership (TCO). Put operating costs into $c$ and the dedicated chip looks far better. Counted per unit, however, what decides the break-even count is mostly the difference in acquisition price; the difference in electricity costs is only a few percent (solution to Exercise 4). Electricity becomes a large sum only in totals at the scale of tens of thousands of units running for several years.
3. Bargaining power: having your own chip gives you option value in price negotiations with commercial GPU vendors. Even if you never use it, its existence is worth something.

Try it yourself: with these three terms added, explain "why hyperscalers build their own chips and most startups do not." The heart of the answer is the difference in scale of $N$ and $C_{\text{NRE}}$, and whether the workload is stable.

The flexibility-efficiency spectrum

The FPGA is an interesting middle point. Its hardware can be reconfigured, so it needs no mask set, but it implements the same function with far more area and power than an ASIC. It suits low-volume, high-mix products and early stages when the algorithm is still in flux.

6. The roofline model — the shared tool of this book

Definition

The performance ceiling is the minimum of two constraints.

$$P_{\text{attainable}}(I)=min(P_{\text{peak}},B\times I)$$

• $P_{\text{peak}}$ : the chip's peak compute rate [FLOP/s]
• $B$ : memory bandwidth [byte/s]
• $I$ : arithmetic intensity [FLOP/byte] = (operations performed) / (bytes moved from memory)

Plotted on log-log axes, a line of slope 1 (the bandwidth limit) meets a horizontal line (the compute limit) to form the shape of a roof. The point where it bends is the ridge point, $I^{*}=P_{\text{peak}}/B$.

Ridge points of real chips

Caution: vendor material also lists a figure twice as high with structured sparsity. The values above are dense. Always check which one you are looking at on a spec sheet.

Capacity also differs between chip and system. NVIDIA's Blackwell Architecture Technical Brief (V2.1) gives "Up to 192 GB" as the GPU memory of the HGX B200. The user guide and datasheet for the DGX B200, the finished server, give 1,440 GB in total for 8 GPUs, which works out to 180 GB per GPU. When you plug capacity into a calculation, state whether it is the chip's published value or a system specification. Bandwidth also varies by document. Technical Brief V2.1 gives the bandwidth of the same HGX B200 GPU as 7.7 TB/s, and the DGX B200 datasheet gives 64 TB/s in total for 8 GPUs (8.0 TB/s per GPU by calculation). The tables and figures in this book use the chip values from the Technical Brief (192 GB, 7.7 TB/s), the same source as the capacity. Calculated with 8.0 TB/s, $I^{*}\approx 281$, so the conclusion below, that it sits near 300, holds either way. The compute figure of 2,250 TFLOP/s is half of the sparse 4.5 PFLOPS in the same table (the table's dense column gives 2.2).

What to notice: from one generation to the next, $I^{*}$ barely moves from around 300. When compute grew 2.3 times, bandwidth was also raised 2.3 times, which means the balance was only just kept. Getting that bandwidth took enormous spending, from HBM3 to HBM3E and on to HBM4. That is the subject of Session 3.

Placing LLM workloads on the roofline

(a) Training / prefill

The tokens of sequence length $S$ and batch $B_{\text{sz}}$ are processed all at once, so this becomes a large GEMM. When a weight matrix $W\in {ℝ}^{n\times n}$ is applied to $M$ tokens:

• Computation ≈ $2M{n}^2$ FLOP
• Memory traffic ≈ $2n^2$ bytes (fp16 weights loaded once)
• Arithmetic intensity $I\approx M$ (when activation traffic is left out of the count)

Leaving out the activations is an approximation. If the $M\times n$ input is read from HBM and an output of the same size is written back, the traffic becomes $2n^2+4Mn$ bytes and $I=Mn/(n+2M)$. This value cannot exceed the smaller of $M$ and $n/2$. So $I\approx M$ holds when $n$ is much larger than $M$, or when kernel fusion keeps the activations on chip. In LLMs $n$ is in the thousands or more ($n=\mathrm{8,192}$ and $M=\mathrm{4,096}$ give $I=\mathrm{2,048}$), so whichever way you count, $I\gg I^{*}$ → compute-bound. The GPU's compute units are well filled.

(b) Inference / decode

Tokens are generated one at a time. Even if a batch of $B_{\text{sz}}$ sequences is processed together, each step handles one token per sequence.

• Computation ≈ $2B_{\text{sz}}{n}^2$ FLOP
• Memory traffic ≈ $2n^2$ bytes (the weights must be read again in full)
• Arithmetic intensity $I\approx B_{\text{sz}}$

With fp16 weights, a batch below 300 is memory-bound. If you serve with a batch of 32, $I\approx 32$, so you use only about $32/295\approx 11\%$ of the theoretical performance. The other 89% of the compute units sit idle waiting for memory.

The figure below shows where the workload falls under the roof when decoding runs on an H100 with fp16 weights and a batch of 32. In the screen edition you can change the chip, phase, batch, weight precision and model size and see when the point crosses the ridge point.

The ceiling on decoding speed — a rule of thumb counted in bandwidth

For a model with $P$ weights (parameters) at $b$ bytes per parameter, the maximum number of tokens generated per second is set by bandwidth alone.

$$\text{tokens/s}\lesssim \frac{B}{P·b}\times B_{\text{sz}}$$

Example: to run a 70B model in fp16 ($b=2$) on one H100 with a batch of 1, the weights alone are 140 GB and do not fit in 80 GB. Split across 2 cards, the total bandwidth is 6.7 TB/s, so the theoretical ceiling is $6.7\text{e}12/(1.4\text{e}11)\approx 48$ tokens/s. This rule of thumb assumes that when the weights are spread over $N$ chips, each chip reads its share at the same time, so the bandwidth adds up $N$ times. Quantized to INT8 ($b=1$), the weights are 70 GB and only just fit on one card. Drop to one card and the bandwidth also falls to 3.35 TB/s, so the ceiling stays at about 48 tokens/s; keep both cards and it doubles to about 96 tokens/s.

This formula counts only the bandwidth side, so it holds only when the arithmetic intensity $I$ is below the ridge point $I^{*}$. If only the weights are read at $b$ bytes and the computation runs at the BF16 peak, then $I=2B_{\text{sz}}/b$, and the batch at which you cross into compute-bound is $B_{\text{sz}}=(b/2)I^{*}$. For the H100 that is about 295 in fp16, about 148 in INT8 and about 74 in INT4. If the computation also runs at the same low precision and the peak rises with it, the boundary moves back to around the BF16 $I^{*}$ (solution to Exercise 3). When compute-bound, the ceiling is set by the $2P$ FLOP that one token takes (for the whole batch, $\text{tokens/s}\lesssim N{P}_{\text{peak}}/(2P)$, where $P_{\text{peak}}$ is the peak at the precision actually used). The traffic from reading the KV cache at every step is also left out, so the longer the context, the further real speed falls below this ceiling. Translated into a metric users feel, for a single sequence this formula gives a lower bound $P·b/B$ on the time between output tokens (TPOT). The time to the first token (TTFT) is set not by this formula but by prefill ("Conditions for a fair comparison" at the end of section 8).

What this one rule of thumb explains: why quantization is essential in inference, why you try to grow the batch, why "do the weights all fit in memory?" is decisive, and why the HBF (high-capacity memory) of Session 3 is being discussed.

Every way to raise arithmetic intensity (summary)

Worth pinning down: in this list, the software items all raise $I$, and half of the hardware items raise $B$. Once you see the picture of two camps attacking different terms of the same inequality, papers and articles become much easier to read.

7. Software lock-in — a moat outside the technology

Even when hardware performance is similar, software decides what actually gets adopted.

• The CUDA ecosystem: optimized kernels built up over many years, such as cuDNN, cuBLAS, NCCL and TensorRT, and extensive library support.
• The alternatives: AMD ROCm, Intel oneAPI, Google XLA/JAX, OpenAI Triton, the compiler stack of PyTorch 2.x. Attempts like Triton to describe kernels in a hardware-neutral way could weaken lock-in. A compiler stack captures the model as a computation graph, lifts it into an intermediate representation (IR), and then lowers it step by step into representations closer to the hardware to produce kernels. According to the PyTorch 2.14 documentation, in torch.compile TorchDynamo captures the graph from Python code, and the default compiler, TorchInductor, generates code for a range of accelerators and backends. On NVIDIA, AMD and Intel GPUs, TorchInductor builds its kernels on OpenAI Triton.
• Why hyperscalers can build their own chips: they control their own workloads, so they only have to fit the compiler stack to their own models. Not needing to build a general-purpose software ecosystem is their decisive advantage.

Operator coverage and fallback

Here is a common reason why the hardware is good but the model is slow. Accelerator libraries do not cover every operator. When the runtime meets an operator it does not support, it hands just that node to the CPU (fallback). The input then has to be moved down from device memory to host memory, and the result the CPU computes has to be moved back up to the device; before and after each transfer, device and host wait for each other. Even if the operation itself is small, this round trip adds to the latency of the whole graph.

Two widely used runtimes document this behavior. ONNX Runtime takes a list of execution providers in order of priority. Given ['CUDAExecutionProvider', 'CPUExecutionProvider'], it runs on CUDA the nodes CUDA can handle and runs the rest on the CPU. PyTorch's Apple GPU backend (MPS) sends operations that MPS does not support to the CPU when the environment variable PYTORCH_ENABLE_MPS_FALLBACK is set to 1 (PyTorch 2.14 documentation).

How expensive the round trip is becomes clear when you set the widths of the two paths side by side. The NVIDIA H100 spec sheet lists PCIe Gen5 for the H100 SXM at 128 GB/s and GPU memory bandwidth at 3.35 TB/s. The H100 white paper states that 128 GB/s is the total for both directions, 64 GB/s each way. Dividing the official figures gives about 26 times; counting one direction, about 52 times. A single fallback moves data that would have stayed inside HBM twice through a path more than an order of magnitude narrower. The claim that lock-in really consists of accumulated kernels (appendix B, question 4) takes a concrete shape here. A new chip with a short operator list can be slowed by fallback even if its arithmetic units are fast.

Case — DeepSeek-V3, hiding communication behind computation

The DeepSeek-V3 technical report (arXiv:2412.19437) gives an example of software making up for hardware constraints. According to the report, the model was trained on a cluster of 2,048 H800s, and full training (2,664K GPU hours for pre-training, 119K for context-length extension, 5K for post-training) took 2.788M H800 GPU hours. The report assumes a rental price of USD 2 per GPU hour to arrive at USD 5.576 million, and in the same paragraph states that this figure covers only the official training and excludes the costs of prior research and ablation experiments on architecture, algorithms and data. So the figure must not be read as the cost of developing the model.

What matters technically is how communication was handled. The report designed DualPipe, an algorithm that overlaps the computation and communication phases of the forward and backward passes in pipeline parallelism, and set aside 20 of the H800's 132 SMs to handle cross-node expert (MoE) communication. That works out to about 15% of the compute units. The report itself notes that this approach limits computational throughput, and it suggests to chip designers that communication be moved off the SMs onto dedicated hardware. It is a case of hiding cross-node communication, an item outside the roofline (appendix B, question 1), behind computation by scheduling, and of giving up part of the compute units in return.

8. Common misconceptions

Misconception 1: "High FLOPS means a fast chip" → If the workload's arithmetic intensity is below the ridge point, FLOPS mean nothing. In LLM inference, the bandwidth figure is the better predictor.

Misconception 2: "You can compare TOPS/TFLOPS figures as they are" → They differ by 4–8 times depending on precision (FP32/BF16/FP8/INT8/INT4) and on whether sparsity is applied. Match the conditions before comparing. Even matching precision and sparsity is not enough — see "Conditions for a fair comparison" at the end of this section.

Misconception 3: "GPU utilization at 90% means it is well used" → The utilization reported by nvidia-smi is "the share of time a kernel is running," not a sign that the compute units are full. A kernel spinning while it waits for memory also shows as 100%. The real metric is achieved FLOPS against theoretical FLOPS, such as MFU (Model FLOPs Utilization). When kernels are very short there is a trap on the other side, too. The overhead of launching a kernel and waiting for it becomes larger than its run time, and the GPU sits idle between kernels (the CUDA Graphs example in appendix B, question 1).

Misconception 4: "Quantization makes computation faster" → The throughput of the arithmetic units does rise, but in inference most of the gain you notice comes from less memory traffic.

Misconception 5: "An ASIC is always more efficient than a GPU" → Only when the workload is fixed. When matrices that do not fit the array size or new operators (for example, a new attention variant) come in, utilization drops sharply.

Misconception 6: "NPUs and GPUs are the same thing at different sizes" → Their objective functions differ (throughput vs. performance per watt). There are many points where design decisions go in opposite directions.

Conditions for a fair comparison — what to match besides precision

The speed of an inference service cannot be captured in one number. The latency a user feels splits into three. TTFT (time to first token) is the time from sending the request until the first token comes out, and it is governed by prefill, which processes the whole input at once. TPOT (time per output token) is the average interval between output tokens, governed by the decoding ceiling of section 6. P99 is the tail latency below which 99% of requests finish. Even with a good average, a long tail means the service-quality promise cannot be kept.

That is why MLPerf Inference, the industry-standard benchmark, ties several conditions together when it compares chips. In the data-center Llama2-70B task of the MLCommons inference rules (the master version checked on October 8, 2026, last modified August 20, 2026), the Closed division requires the same model and preprocessing as the reference implementation (quantization calibration is allowed, retraining is not) and fixes the dataset as OpenOrca. Accuracy must be at least 99.9% of the FP32 reference ROUGE scores, and the latency constraints are TTFT 2,000 ms and TPOT 200 ms in the Conversational category, TTFT 450 ms and TPOT 40 ms in the Interactive category. In the Server and Interactive scenarios, where requests arrive at random following a Poisson distribution, latency is measured at the 99th percentile. A chip that raised its throughput with lower precision also has to clear the accuracy floor and the latency constraints for its result to count. One line of a TOPS table tells you none of this. The rules change with every round, so when you cite them, state the version.

9. Quantitative exercises

Exercise 1 — Computing the ridge point Pick any accelerator from a spec sheet, find the dense $P_{\text{peak}}$ and $B$, and compute $I^{*}$. Compare three generations and look at the trend in $I^{*}$.

Exercise 2 — The decoding ceiling For 7B / 70B / 405B models served in fp16, INT8 and INT4, compute the theoretical maximum tokens/s at a batch of 1 on a given accelerator. Which combinations meet a real service latency requirement (for example, 30 tokens/s)?

Exercise 3 — The KV cache budget With sequence length $S$, layers $L$, total KV head dimension $d$ (number of heads storing K and V × dimension per head; query heads are not counted) and batch $B_{\text{sz}}$, the size of an fp16 KV cache is roughly $2·L·S·d·B_{\text{sz}}·2$ bytes. Find the largest batch possible for a given memory capacity, and place the arithmetic intensity at that batch on the roofline. How the capacity constraint limits bandwidth efficiency comes out.

Exercise 4 — ASIC break-even Using assumed values (NRE USD 100 million, ASIC unit price USD 3,000, GPU unit price USD 30,000, 3 years of electricity included), find $N^{*}$ and discuss whether it is achievable if the model's life is 2 years.

10. Discussion questions

1. What does the roofline model fail to explain? (Latency, the cache hierarchy, communication, kernel launch overhead and so on. Knowing a model's limits is half of knowing the model.)
2. Is it chance or design that the ridge point stays near 300 across generations? What happens if it moves away in either direction?
3. What effect does MoE (mixture of experts) have from the roofline's point of view? How should we assess a structure in which total parameters grow but active parameters shrink?
4. How long can software lock-in last? What changes as intermediate layers such as Triton and XLA mature?
5. If you were designing a new chip only for inference, what would you do differently from a training chip? (Hint: where would you put the ridge point?)

11. Further study

• The roofline original: Williams, Waterman & Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures", CACM 52(4), 2009 — DOI 10.1145/1498765.1498785
• The source of the energy figures: Horowitz, "Computing's Energy Problem (and what we can do about it)", ISSCC 2014 keynote — DOI 10.1109/ISSCC.2014.6757323. The two 32-bit memory values in the section 1 table come from Figure 1 of Han, Pool, Tran & Dally, "Learning both Weights and Connections for Efficient Neural Networks", NIPS 2015 — arXiv:1506.02626
• TPU architecture: Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit", ISCA 2017 — the paper that sets out the design rationale of TPU v1 directly. If you read only one paper alongside this session, make it this one. arXiv:1704.04760
• Accelerator design textbook: Sze, Chen, Yang & Emer, "Efficient Processing of Deep Neural Networks" (Synthesis Lectures, 2020) — the standard treatment of the dataflow taxonomy (weight stationary / output stationary and so on). DOI 10.1007/978-3-031-01766-7
• Kernel-level optimization: Dao et al., the "FlashAttention" series — the leading example of raising $I$ through kernel fusion. The first paper is arXiv:2205.14135
• GPU architecture: NVIDIA architecture white papers (H100 white paper, Blackwell Architecture Technical Brief) and the memory-hierarchy chapters of the CUDA Programming Guide
• Kernel launch overhead: NVIDIA technical blog, "Getting Started with CUDA Graphs" (2019) — an example that measures how batching short kernels into one launch works and what it gains
• The rules of comparison: MLCommons MLPerf Inference rules — accuracy floors and TTFT/TPOT constraints for each task

Self-study appendix A — solutions to the quantitative exercises

These are the solutions to the exercises in section 9. Solve them yourself first, then open them. The numbers come from public spec sheets, and all are dense (no sparsity) values.

Solution to Exercise 1 — Computing the ridge point

Solution to Exercise 2 — The decoding ceiling

Solution to Exercise 3 — The KV cache budget

Solution to Exercise 4 — ASIC break-even

Self-study appendix B — leads for the discussion questions

Self-study appendix C — key terms

Self-study appendix D — self-assessment

• ☐ I can write $P_{\text{attainable}}=min(P_{\text{peak}},B\times I)$ without looking and say what each term is → section 6 ★
• ☐ Given any chip's spec sheet, I can compute $I^{*}$ within 30 seconds → Exercise 1 ★
• ☐ I can explain why prefill is compute-bound and decoding memory-bound, using the $I$ of each → section 6
• ☐ Using the decoding-ceiling formula, I can give two reasons why "70B fp16 on one H100" is impossible → Exercise 2
• ☐ I can explain why quantization improves both the ceiling and capacity → Exercise 2
• ☐ I can set up the KV cache size formula and explain how capacity limits bandwidth efficiency → Exercise 3 ★
• ☐ I can tell ways of raising arithmetic intensity from ways of raising bandwidth and name three of each → the last table in section 6
• ☐ I can show by calculation that in ASIC break-even acquisition cost, not electricity, dominates → Exercise 4
• ☐ I can name three things the roofline cannot explain → appendix B

If you do not pass the three ★ items, do not move on to Session 3. All of Session 3 is the answer to "so how do we raise $B$?"

Before the next session

The conclusion of this session fits in one sentence. For work whose arithmetic intensity falls short of the ridge point, small-batch decoding above all, performance is set by bandwidth, not by compute. Work with high arithmetic intensity, such as prefill and training, is still compute-bound.

That leads naturally to the next questions.

Sources

Introductory material chosen by the study curriculum (the "AI chips (compute semiconductors)" item)

Verified (2026-08-02)

• Horowitz (ISSCC 2014) energy figures — widely cited standard reference values (which values appear in the original talk is covered in the 2026-10-08 item below). Because they are for a 45nm process, use them to grasp the size of the ratio (two orders of magnitude), not as absolute values
• NVIDIA H100 SXM: BF16 dense 989 TFLOP/s, HBM3 80 GB, 3.35 TB/s — manufacturer's published specifications
• NVIDIA B200: BF16 dense 2,250 TFLOP/s, HBM3E 192 GB, 7.7 TB/s — manufacturer's published specifications (Blackwell Architecture Technical Brief V2.1, Table 3, HGX B200). 192 GB and 7.7 TB/s are the chip's published (maximum) values. The DGX B200 system has 180 GB and 8.0 TB/s per GPU (64 TB/s total ÷ 8). The system specifications differ — see the 2026-10-08 item below
• TPU v1 $256\times 256$ MAC array — Jouppi et al., ISCA 2017, original paper

Verified (2026-10-08) — all of the following were checked by opening the primary sources directly.

• Chip vs. system distinction for B200 memory capacity — "Up to 192 GB HBM3e" per HGX B200 GPU in the NVIDIA Blackwell Architecture Technical Brief V2.1; 1,440 GB total for 8 GPUs (180 GB per GPU by calculation) in the DGX B200 User Guide and the DGX B200 datasheet (English edition: the Technical Brief wording rechecked against the PDF, 2026-10-22)
• Power boundaries — H100 SXM maximum TDP 700 W (H100 spec sheet), DGX H100 maximum 10.2 kW (DGX H100/H200 User Guide), DGX B200 maximum about 14.3 kW (DGX B200 datasheet and user guide). The server-boundary recalculation in Exercise 4 was redone with these values and the assumed values of Exercise 4
• H100 SXM PCIe Gen5 128 GB/s and GPU memory 3.35 TB/s — official figures on the H100 spec sheet. The H100 white paper V1.04 states that 128 GB/s is the total for both directions (64 GB/s each way)
• Fallback behavior — the description of the priority list in the ONNX Runtime execution providers documentation, and PYTORCH_ENABLE_MPS_FALLBACK in the PyTorch 2.14 MPS environment variables documentation
• The structure of torch.compile (TorchDynamo, TorchInductor, Triton) — PyTorch 2.14 torch.compiler documentation
• Kernel launch overhead of 2.9, 9.6, 3.8 and 3.4 μs — the text of the NVIDIA technical blog post "Getting Started with CUDA Graphs" (2019-09-05). Tesla V100, CUDA 10.1
• Intel AMX tile configuration (8 tiles, 8 KB in total, at most 16 rows × 64 bytes) and TMUL — Intel AMX code-sample article
• The accuracy floor (99.9% of the FP32 reference) and TTFT/TPOT constraints of the MLPerf Inference Llama2-70B task, the definitions of TTFT and TPOT, and the 99th percentile in the Server and Interactive scenarios — MLCommons inference rules, master version (last modified 2026-08-20, commit ff7edba)
• DeepSeek-V3's 2,048 H800s, 2.788M GPU hours (2,664K + 119K + 5K), USD 5.576 million at an assumed USD 2 per hour, the caveat excluding prior research and ablation experiments, DualPipe, and 20 of 132 SMs — text of arXiv:2412.19437 v2
• Systolic-array beat counts and utilization at TPU v1 size (0.2%, about 89%) — recalculated with the same formula as the interactive figure, $M/(M+K+N-2)$
• Energy table in section 1 — checked in a public copy of Horowitz's paper (ISSCC 2014, paper 1.1), Figure 1.1.9, "Rough energy costs for various operations in 45nm 0.9V": 8-bit integer add 0.03 pJ, 32-bit floating-point add 0.9 pJ and multiply 3.7 pJ, cache (64-bit access) 8 KB 10 pJ, 32 KB 20 pJ, 1 MB 100 pJ, DRAM 1.3–2.6 nJ. The 32-bit SRAM 5 pJ and DRAM 640 pJ are not in this figure; they are in Figure 1 ("Energy table for 45nm CMOS process", citing Horowitz) of Han et al. (2015), arXiv:1506.02626 v3
• Memory coalescing with 32-byte transactions, 4 consecutive bytes × 32 threads = 128 bytes = 4 transactions, 32 at worst — CUDA Programming Guide §2.3.4.1. §2.3.3.2 and §2.3.3.6 of the same chapter describe the L1 and L2 caches and shared memory
• H100 SXM INT8 tensor core 3,958 TOPS (with sparsity; dense is half that, 1,979, 2 times the BF16 dense 989) — H100 spec sheet. The spec sheet gives no INT4 value (the INT4 peak in Exercise 3 is an assumption)

Unconfirmed · reserved

• Specific amounts for ASIC NRE and mask-set costs — they vary widely by node, company and contract and are often not disclosed. The text mentions them only by order of magnitude, and the exercise states that they are assumed values
• AI accelerator shares by company — not asserted, as in Session 1
• Unit prices and electricity costs in the section 5 exercise — all illustrative assumed values, not real market prices

Session 3

AI Memory — DRAM · HBM · HBF

Learning objectives

1. Explain why the different device structures of SRAM, DRAM and NAND produce their characteristic speed, capacity and cost.
2. Explain why DRAM latency has barely improved in 20 years.
3. Reconstruct the structure of HBM from the question "why does it look like this?"
4. Calculate the multiplicative structure of stack yield and explain the cost structure of HBM.
5. Distinguish the bottleneck HBF targets from the one HBM targets.

0. Self-study guide — for readers working alone

What carries over from the previous session

Prerequisite check

Reading order (110 minutes in total recommended)

The one sentence that runs through this session

Bandwidth and capacity are different constraints, and different technologies solve them. Bandwidth: HBM (close and wide). Capacity: HBF and tiering (farther away but large). Mix the two up and this session will confuse you throughout.

A common sticking point — why "wide and slow"?

Intuitively, raising pin speed looks easier. Isn't making one lane twice as fast simpler than planting 1,024 pins?

No. Raising the transfer rate makes energy per bit grow nonlinearly (equalization circuits, clock recovery, driver strength), and signal-integrity problems get sharply worse. Build 4 TB/s of bandwidth from 1,024 pins at 32 Gbps and the interface alone can eat close to 100W per stack (96W at 3 pJ/bit). With 8 stacks on one accelerator, the memory interface alone takes about 770W, a second heat source on par with the GPU's own power (700–1,000W). How much the temperature actually rises depends on the area over which that heat spreads and on the cooling, but it is hard to give the memory side a power budget of that size. So HBM chose to use very many slow pins, and the price was that it needed room to plant them, which brought in stacking, TSVs and interposers. Exercise 2 checks this logic with numbers.

1. Getting the intuition — the memory hierarchy is a map of physical compromises

Readers who have studied computer architecture will know the hierarchy "registers → L1 → L2 → L3 → DRAM → SSD." Such courses usually do not cover why the hierarchy splits into those sizes and speeds. The answer lies in the device structure.

One-sentence summary: the number of devices it takes to hold one bit sets the character of the whole hierarchy. As transistors per bit fall from 6 for SRAM → 1 for DRAM (plus one capacitor) → 1/3–1/4 for NAND (3–4 bits per cell), memory gets cheaper and denser but slower and harder to handle. NAND then stacks cells vertically in hundreds of layers, multiplying the bits held in the same area once more. Cutting the device count and stacking layers are separate sources of density.

The hierarchy as seen from an AI accelerator (as of October 2026)

Inside an AI server the same devices sit in the order below. The higher the level, the shorter the latency; the lower, the larger and cheaper. Looking at stack bandwidth alone, HBF can exceed host DRAM, but NAND's long read latency puts it a level below DRAM. In its keynote at FMS in August 2026, SK hynix presented the memory hierarchy as four levels: HBM, DRAM, NAND (HBF) and SSD. This table adds on-chip SRAM on top of that order.

The order is by speed and capacity, not by physical distance. HBF sits below DRAM, yet it is meant to be placed close to the accelerator as HBM is. As an analogy, SRAM is the top of your desk, HBM the bookshelf within arm's reach, DRAM the stacks in the same room, and SSD the basement archive. HBF is a large new set of shelves placed next to the desk: close by, but slower to pull a book from than the stacks. The analogy captures only distance, retrieval speed and size. NAND's wear with every write and the fact that it can be erased only block by block have no counterpart in the book analogy.

2. The physics of DRAM — why it is slow and why it does not get faster

What a read actually does

A DRAM cell stores 1 bit as the presence or absence of charge on a very small capacitor. To read it:

1. Wordline activation — turn on the access transistors of the row
2. Charge sharing — the charge on the cell capacitor flows out onto the bitline and shifts the bitline voltage very slightly. Cell capacitance is far smaller than bitline capacitance, so the change is only on the order of tens of mV
3. Sense amplification — compare against a reference voltage and amplify that tiny difference to full swing
4. Reading = destroying — the charge has left, so the original value must be written back (restore)
5. Precharge — return the bitline to the midpoint voltage for the next access

What the timing parameters mean

This procedure is exactly what DRAM timing is made of.

The key fact: these are analog physical processes governed by charge and capacitance, so process scaling does not speed them up much. If anything, smaller cells store less charge and sensing gets harder.

Twenty years of asymmetry: DRAM bandwidth has grown dozens of times over, but latency has stayed nearly the same. Bandwidth can be raised through parallelism and interface speed, but latency is set by the physical process in a single cell. This asymmetry pushed all of computer architecture toward "hiding latency" (prefetching, multiple banks, the massive thread parallelism of GPUs).

The refresh tax

Charge leaks, so cells must be read and rewritten periodically. During refresh the bank cannot be accessed, which eats into bandwidth and power. The burden grows with capacity, making it a standing problem in high-capacity DRAM design.

It is easy to confuse this with step 4 above (restore after a read). Restore is caused by an access; refresh is caused by time. Even a cell nobody reads must be refilled once within a fixed period.

Worked example — how large is the tax? Take the Micron 16Gb DDR4 data sheet (Rev. H, 2021). At or below 85°C, 8,192 REFRESH commands must be issued within 64ms, so the command interval tREFI is 7.8μs (64ms ÷ 8,192 = 7.8125μs). Between 85 and 95°C it is 8,192 within 32ms, so tREFI halves to 3.9μs. The time one command ties up the chip, tRFC1, depends on density. The fraction of time the chip is tied up in refresh is roughly tRFC/tREFI.

While density quadrupled, the tax more than doubled, and it doubles again when the chip runs hot. These ratios divide tRFC1 by 7.8125μs and 3.90625μs. It is a simple calculation that assumes the whole chip can do nothing else during refresh.

3. Two ways to raise bandwidth, and the dead end

Memory bandwidth is approximately

$$B=(\text{bus width})\times (\text{transfer rate per pin})$$

So there are only two methods.

Think of a road: bus width is the number of lanes, and the per-pin transfer rate is how fast the cars go. To get more cars past per hour, you add lanes or make the cars go faster. The analogy misses something. On a road, faster cars do not make sending one car steeply more expensive, but a memory pin spends steeply more energy per bit (pJ/bit) the faster it sends. That cost is why the two paths below part ways.

Path ①: raise the per-pin transfer rate (the GDDR path)

• Toggle the signal faster. GDDR6X and GDDR7 go as far as multilevel signaling (PAM3/PAM4).
• How fast has it become? The JEDEC GDDR7 standard (JESD239, published 2024-03-05) uses PAM3, which sends 3 bits in 2 cycles using three voltage levels, and reaches up to 192 GB/s per device. For a 32-bit device that works out to 48 Gbps per pin, 6 times the HBM4 standard pin speed of 8 Gbps. HBM does not use slow pins because pin speed cannot be raised.
• The limit: raising the frequency makes signal integrity deteriorate sharply. Loss, reflection and crosstalk in board traces all worsen with frequency, and the equalization and serialization circuits that compensate eat power. The decisive problem is that energy per transferred bit (pJ/bit) goes up.
• At data-center scale that power is both operating cost and heat, so this path alone cannot get there.

Path ②: widen the bus (the HBM path)

• A DDR5 channel is 64 bits and GDDR is 32 bits per device, while an HBM stack is 1,024 bits (through HBM3), and HBM4 is 2,048 bits.
• The same bandwidth can be had at a much lower frequency, so pJ/bit falls substantially.
• The limit: to pull out thousands of wires, the wires need somewhere to go. An ordinary PCB substrate cannot reach that wiring density, and a package does not have enough pins either.
• A caveat — there is also a standard that mixes the two paths. SPHBM4 (JESD330-4), published by JEDEC on July 13, 2026, uses the same DRAM dies as HBM4 but changes the base die, cutting the data signals from 2,048 to 512 and using 4:1 serialization to reach the same aggregate bandwidth. With fewer pins the bump pitch can be relaxed, so the stack can be mounted on an ordinary organic substrate instead of a silicon interposer, and the memory can sit farther from the SoC. "A wide bus requires an interposer" holds when all 2,048 signals are used; pay the price of running each signal four times faster and you can move partway back toward path ①. What that price is in pJ/bit is not in the press release.

The answer that emerged

1. To pull out a very large number of wires → you need a medium that allows extremely dense wiring → a silicon interposer
2. To connect several dies inside a stack → TSVs (through-silicon vias)
3. Shorter distance means less power and latency → attach it right beside the GPU (2.5D packaging)

At that moment the memory problem became a packaging problem. That is why Session 6 is the most important session in this book.

4. HBM structure in detail

Physical layout

What each element does

• Stacked DRAM dies: capacity comes from height rather than area. 8-high, 12-high, 16-high.
• TSVs: vertical wiring that runs through each die. They bind the whole stack into one wide bus.
• Base die: a layer made in a logic process, not a DRAM process. It holds the PHY (physical-layer interface), test circuits and control logic. With HBM4, making this base die in a foundry logic process has become reality. Samsung Electronics used a 1c DRAM core and a 4nm logic process in the HBM4 it began shipping in volume in February 2026, and its May 2026 press release on HBM4E samples stated that the base die is "Samsung Foundry's 4nm logic base die" and that HBM4E uses "the same combination of core and base die" as HBM4. This is where the line between memory makers and foundries blurs.
• Die-to-die bonding — microbumps and what comes next: HBM in volume production today joins dies with microbumps and fills the gaps between them. SK hynix used MR-MUF (pouring a liquid protective material between the stacked chips and curing it) for HBM4 (announced 2025-09), and Samsung Electronics used TC-NCF (laying a non-conductive film on each layer and thermocompression bonding it) for its 12-high HBM3E (announced 2024-02). Hybrid bonding, which joins copper to copper directly without bumps, is under development as the next candidate for taller stacks; as of October 2026 we found no manufacturer announcement of volume production of HBM using it. Hybrid bonding itself is covered in Session 6.
• Interposer: the silicon plate that actually draws the thousands of wires between the GPU and the HBM.

Specifications by generation

The standard column gives what the pin speed set by the JEDEC standard can deliver; the product column gives values the manufacturers announced. As of October 2026, volume HBM4 exceeds the standard pin speed.

• SK hynix announced on September 12, 2025, that it had completed HBM4 development and readied mass production: 2,048 I/O, over 10 Gbps per pin (above the JEDEC standard of 8 Gbps), 1b DRAM (the fifth generation of the 10nm class), and the Advanced MR-MUF process.
• Samsung Electronics announced volume shipments of HBM4 on February 12, 2026. It runs at 11.7 Gbps per pin, can be pushed to 13 Gbps, and comes as a 12-high stack of 24–36 GB. Multiply 11.7 Gbps by 2,048 bits and you get about 3.0 TB/s; at 13 Gbps, about 3.3 TB/s. The press release's "maximum of 3.3 terabytes-per-second (TB/s)" per stack is the 13 Gbps figure.
• The same company's 12-high HBM4E samples (2026-05-29): assume 2,048 bits as for HBM4 and multiply by 14 Gbps, and you get about 3.6 TB/s, matching the press release (which does not state the interface width).
• The standard-column values for HBM2E and HBM3 and the HBM3E row are rough ranges, because pin speed differs by vendor and product. Blank cells were not confirmed from a primary source for this edition.

Main changes in HBM4

• Interface width 1,024 → 2,048 bits (2 times)
• Independent channels 16 → 32, each with 2 pseudo-channels
• Backward compatibility with HBM3 controllers — the interface is defined so that a single controller can handle both HBM3 and HBM4
• DRFM (directed refresh management) introduced — row-hammer mitigation and better reliability (RAS)
• Supports 4- to 16-high stacks, die density 24Gb or 32Gb → 32Gb at 16-high gives 64 GB per cube
• Lower core voltage (1.1V in HBM3/3E → 1.0/1.05V); VDDQ chosen by the vendor from 0.7–0.9V

Note: HBM4 doubled the width instead of raising the frequency sharply. That means it chose "path ②" from section 3 once again. Power efficiency is why this path keeps being chosen.

Why each GPU carries several HBM stacks

Because one stack is short on both bandwidth and capacity. The latest accelerators arrange 6–8 HBM stacks around the GPU die. As one example, NVIDIA said the Rubin GPU, with 12-high HBM4 stacks, delivers up to 288 GB and up to 22 TB/s per GPU (technical blog, 2026-07-21; 2.8 times Blackwell's bandwidth). The post does not give the number of stacks. Assuming 36 GB stacks, there are 8, which gives 2.75 TB/s per stack and about 10.7 Gbps per pin — an estimate worked backward, nothing more. But adding stacks:

• enlarges the interposer → the interposer, too, is limited by the lithography exposure field → extension methods such as CoWoS-L are needed
• makes the package bigger and raises heat density
• makes packaging capacity the ceiling on AI chip shipments (what the supply-chain bottleneck in Session 1 actually is)

5. Why HBM is expensive — yields multiply

For readers trained in statistics, this is the most intuitive section.

The basic calculation

Let $y$ be the yield of one DRAM die, and stack $n$ of them. If dies are simply picked at random and stacked, the probability that the whole stack works is

$$Y_{\text{stack}}=y^n\times {\prod }_{i=1}^{n-1}{y}_{\text{bond},i}$$

where $y_{\text{bond}}$ is the yield of each bonding step.

Numerical example ($y=0.99$, bonding yield 0.995)

The stack-yield column multiplies the unrounded values, so multiplying the two printed columns may differ in the last digit.

Even with an excellent 99% per die, a 16-high stack throws away 1 stack in 5. And when a stack is thrown away, its good dies go with it. If one die is bad, the other 15 good dies are discarded; if all 16 dies are good but one bond fails, all 16 are discarded. This is the heart of HBM's price structure.

Countermeasures

1. KGD (known good die) screening — test each die fully before stacking and use only good ones. This pushes $y$ in the formula above toward 1. But a full test at wafer level needs separate test structures and time, and that is itself a cost.
2. Redundancy — provide spare rows, columns and TSVs to replace defects. DRAM has traditionally been good at repair.
3. Better bonding yield — one of the reasons hybrid bonding is being considered as the next candidate.

The cost structure in brief

HBM costs several times as much as ordinary DRAM of the same capacity because of a combination of (a) low stack yield, (b) extra steps such as TSV formation, thinning and bonding, (c) the cost of the interposer and advanced packaging, and (d) a market in which supply is limited to three companies.

For discussion: which of these terms improves over time, and which remains structurally?

6. HBF (high bandwidth flash) — capacity, another bottleneck

Why it appeared

Look again at the decoding ceiling from Session 2.

$$\text{tokens/s}\lesssim \frac{B}{P·b}\times B_{\text{sz}}$$

Raising the batch size $B_{\text{sz}}$ raises throughput, but a larger batch needs memory capacity to hold the KV cache. And as the model itself grows, it needs capacity to hold the weights. Bandwidth and capacity are separate constraints.

HBM is optimized for bandwidth, but its capacity is expensive (around 24–32Gb per die). NAND, by contrast, is overwhelmingly denser. "What if we stacked NAND like HBM and gave it a wide bus?" is the idea behind HBF.

Status and specifications

• On August 6, 2025, Sandisk and SK hynix announced a collaboration (memorandum of understanding) to standardize HBF.
• On February 25, 2026, the two companies began standardization work as a dedicated workstream within the OCP (Open Compute Project).
• On August 4, 2026, the two companies released the first HBF standard specification through the OCP (SK hynix announcement). Two configurations, 8-high and 16-high, give up to 512 GB per stack; bandwidth comes in three grades (Grade 1–3) of about 0.4–3.0 TB/s; the connection to the processor is UCIe, the chiplet interconnect standard. Google and Tenstorrent have joined the consortium. The value for the middle grade could not be confirmed from public material.
• Sandisk's earlier first-generation product target (July 2025 material) was 256Gb (gigabits) per die, a 16-high stack of 512 GB, and 1.6 TB/s read bandwidth, aiming at a footprint, power profile and stack height close to HBM4's. 256Gb is 32 GB, so 16-high gives 512 GB. The 1.6 TB/s is the target for one product, within the grade range of the standard above.
• Schedule: in August 2025, Sandisk set targets of first HBF samples in the second half of 2026 and samples of HBF-equipped AI inference devices in early 2027. Its investor day presentation of August 2026 (Sandisk Investor Day 2026, HBF presentation PDF) put the tape-out of the first HBF memory die and 2027 samples of the first HBF inference product on its roadmap, and did not repeat the target of memory samples in the second half of 2026.
• Technology base: Sandisk's BiCS NAND and CBA (CMOS directly Bonded to Array, a structure that bonds the CMOS circuit wafer directly to the cell-array wafer) + SK hynix's HBM packaging and 3D stacking capabilities.

The structure of the trade — what you give and what you get

The key point: HBF is not something that replaces HBM; it adds one level to the memory hierarchy. The intended arrangement keeps large data that rarely changes (model weights, inactive MoE experts, long-context caches) in HBF and keeps actively updated data in HBM.

Mixing the two levels — a simple model

Suppose that in one decoding step a fraction $f$ of the bytes read comes from HBF and the remaining $1-f$ from HBM. If the two are read one after the other, the time taken is the sum of the two shares, so the effective bandwidth is

$$B_{\text{eff}}=\frac{1}{(1-f)/B_{\text{HBM}}+f/B_{\text{HBF}}}$$

and the capacity available is the sum of the two levels, $C_{\text{HBM}}+C_{\text{HBF}}$.

Worked example. Pair one HBM4 stack (up to 3.3 TB/s as announced by Samsung, 12-high 36 GB) with one HBF stack (512 GB). For HBF bandwidth, use the bottom of the published specification range, 0.4 TB/s, as an assumed value. If only 10% of the bytes read come from HBF ($f=0.1$),

$$B_{\text{eff}}=\frac{1}{0.9/3.3+0.1/0.4}\approx 1.91\text{ TB/s}$$

which is 58% of reading from HBM alone. At the same $f$, assuming the top grade of 3.0 TB/s for HBF gives about 3.27 TB/s, almost no loss. Capacity is 36 + 512 = 548 GB either way. Let the share read from the slow level grow even a little and that level sets the overall speed. So HBF's place is data for which $f$ can be kept small — for example, in MoE, the expert weights not used in the current step: mostly just held, read only now and then (Exercise 4).

This model assumes the two levels are read in sequence. It leaves out reading both at once so the reads overlap, and moving needed data into HBM ahead of time (prefetching), and it folds NAND's long latency into a single bandwidth number. All bandwidth values are announced or assumed values for a single stack.

The figure below shows the worked example above: $f=0.1$, HBM4 at 3.3 TB/s and 36 GB, HBF at 0.4 TB/s (assumed) and 512 GB. In the screen edition you can change $f$, HBM bandwidth and capacity, and HBF bandwidth (within the published range of 0.4–3.0 TB/s).

Open questions — what has not yet been verified

• NAND read latency is far longer than DRAM's. How many real workloads can offset it with bandwidth?
• How will write endurance (wear leveling) be managed at the accelerator level?
• Will the software stack (the policies that decide which data goes on which level) mature?
• How standardization proceeds and whether the technology is actually adopted — as of October 2026 the first standard specification has been released but there are no volume products, and the performance figures are specification ranges and targets. Treat this as "an attempt in progress," not "an established fact." Intel Optane (3D XPoint), an earlier attempt to fill the gap between DRAM and NAND, began winding down in the second quarter of 2022. A candidate for a new level does not necessarily take hold (appendix B).

7. PIM / PNM — taking computation to the memory

In the roofline formula $P=min(P_{\text{peak}},B\times I)$, the approaches so far have raised $I$ (software) or raised $B$ (HBM). A third approach is not to move the data in the first place.

• PIM (processing-in-memory): put simple compute units inside the memory die. Place a multiply-accumulate unit in each DRAM bank and the internal bandwidth (not limited by external pins) can be used directly. Samsung's HBM-PIM and SK hynix's AiM are among the attempts.
• PNM (processing-near-memory): put compute units near the memory (for example, on the base die or a buffer chip).

Why it is not yet mainstream

• A DRAM process is not a logic process. Putting complex logic on the same die worsens cost and yield.
• There is no programming model. The compilers and runtimes that decide which operations to send down to memory are immature.
• It is not general-purpose; the gain is large only for particular operations (for example, embedding lookup and GEMV).

Why it still deserves attention: the move to make the base die in a foundry logic process, as in Samsung's HBM4 (section 4), is a change that opens the door to PNM. The boundary between memory and compute is physically blurring.

8. Common misconceptions

Misconception 1: "HBM is just fast DRAM" → The cells themselves are not very different from ordinary DRAM. What differs is the interface width and the packaging. HBM is closer to a packaging technology than to a memory technology.

Misconception 2: "HBM is always better than GDDR" → It wins on absolute bandwidth and power efficiency, but it costs far more, its capacity is harder to scale, and it is in short supply. There is a reason gaming GPUs still use GDDR.

Misconception 3: "Increase HBM capacity and the memory problem is solved" → Capacity and bandwidth are different problems. And raising bandwidth does not settle things either. If $P_{\text{peak}}$ stays the same and $B$ grows, the ridge point $I^{*}=P_{\text{peak}}/B$ comes down (for example, the H100 from Session 2, 989 TFLOP/s ÷ 3.35 TB/s ≈ 295 FLOP/byte, becomes about 148 FLOP/byte if $B$ doubles). When the ridge point comes down, workloads with lower arithmetic intensity also cross over to the compute-bound side, but a workload like decoding, whose arithmetic intensity is below even the new ridge point, stays bound by bandwidth.

Misconception 4: "DRAM keeps getting faster as it scales" → Bandwidth grows, but latency does not improve. If anything, there is pressure in the other direction: cell capacitors shrink and sensing becomes harder.

Misconception 5: "HBF will replace HBM" → Because of its write behavior and endurance, it adds a level rather than replacing one. And as of October 2026 only the first standard specification has been released; there are no volume products.

Misconception 6: "More layers increase capacity at linear cost" → The capacity of one good stack is proportional to its height (die capacity × number of dies). What is not proportional is the cost of getting that stack. Stack without screening and stack yield falls as $y^n$, so the dies needed for one good stack, $n/y^n$, grow faster than the height. Taller stacks also tighten the constraints of heat removal and stack height (package standards).

Misconception 7: "An HBM channel is like a mainframe I/O channel" → Only the name is shared. A mainframe I/O channel is a dedicated processor that handles input and output on the CPU's behalf. An HBM channel is a unit into which the stack's wide bus is divided, each part operating independently. Bus width = number of channels × channel width, so HBM3 is 16 channels × 64 bits = 1,024 bits and HBM4 is 32 channels × 64 bits = 2,048 bits (the channel width is the bus width the standard states divided by the channel count). GDDR has the same structure; in GDDR7, JEDEC increased the independent channels from 2 in GDDR6 to 4.

Misconception 8: "Refresh refills the charge that leaves during a read" → That is restore. Restore is caused by an access (the read); refresh is caused by time. Even cells that are not read must be refilled within a fixed period because of leakage. For DDR4, at or below 85°C, 8,192 REFRESH commands must be issued within 64ms (section 2).

Misconception 9: "Pin speed is stuck at a physical limit and cannot go higher" → It is not that signals cannot be sent faster; it is that sending them faster is expensive. The GDDR7 standard already provides 48 Gbps per pin (192 GB/s per device divided by 32 bits). What holds speed back is not an absolute limit but the channel loss that grows with speed, the pJ/bit driven up by the equalization circuits that compensate for it, and the shorter distance a signal can travel (section 3).

9. Quantitative exercises

Exercise 1 — Stack yield Vary die yield $y$ over 0.95 / 0.98 / 0.995, calculate the stack yield for 8-, 12- and 16-high stacks, and find the number of good dies discarded without KGD screening. The answer to "why do HBM makers invest so much in testing?" comes out in numbers.

Exercise 2 — Bandwidth design To reach a target bandwidth of 4 TB/s with (a) a 1,024-bit bus and (b) a 2,048-bit bus, what must the per-pin transfer rate be? Discuss qualitatively how the difference in pJ/bit between the two affects system power.

Exercise 3 — Allocating the memory budget When serving a given model, how should the available memory be split between weights and KV cache to maximize throughput? A larger batch grows the KV cache; quantization shrinks the weights. Set it up as a problem of finding the optimum.

Exercise 4 — Judging fitness for HBF Set up criteria for judging which workloads would benefit from HBF. (Hint: read/write ratio, reuse distance, latency sensitivity.)

10. Discussion questions

1. Reconstruct the shape of HBM from its physical constraints. If there had been no signal-integrity problem, would HBM exist?
2. How might the move to make HBM4's base die in a logic foundry process (Samsung has done it with its own foundry's 4nm process) change the competitive structure of the memory industry?
3. If the memory hierarchy is subdivided further into SRAM–HBM–DRAM–HBF–SSD, what does software have to do? Can that decision be automated?
4. What must be solved first for PIM to become practical? Is it a hardware problem or a software problem?
5. If the bandwidth race keeps depending on pJ/bit improvements, where is the physical limit? Could optical interconnect be the answer? (Continued in Session 7.)

11. Further study

• Memory systems textbook: Jacob, Ng & Wang, Memory Systems: Cache, DRAM, Disk (Morgan Kaufmann, 2007) — the standard reference on DRAM timing and structure
• DRAM research: lecture materials and papers from the home page of Prof. Onur Mutlu (ETH Zürich) and the SAFARI Research Group — the most systematic body of work on DRAM latency, refresh and PIM, with lecture videos available online. Recommended for readers who want to go deeper into DRAM
• HBM standards: JEDEC JESD235 (HBM), JEDEC JESD270-4 (HBM4) — the standard documents themselves (as of October 2026 the HBM4 standard page carries the revision JESD270-4A). For summaries, the technical blogs of Rambus and Synopsys are accessible
• The signal-speed standard: JEDEC JESD239 (GDDR7) — the current form of path ①, raising pin speed with PAM3
• HBF: Sandisk press release on collaborating with SK hynix on standardization (2025-08-06), SK hynix, first HBF standard specifications (2026-08-04), IEEE Spectrum, "The Memory in Your Thumb Drive Could Fix AI's Big Problem" (2026-07-14), OCP workstream documents
• PIM: ISSCC and Hot Chips presentations on Samsung's HBM-PIM and SK hynix's AiM
• Korean technology trends: ETRI, Electronics and Telecommunications Trends vol. 41 no. 3, "Key Technologies and Market Trends in Artificial Intelligence (AI) Memory Semiconductors" (in Korean)

Self-study appendix A — solutions to the quantitative exercises

Solution to Exercise 1 — Stack yield

Solution to Exercise 2 — Bandwidth design

Solution to Exercise 3 — Allocating the memory budget

The same calculation can be tried in the figure at the end of the solution to Exercise 3 in Session 2. In the screen edition, change the sequence length and precision and watch how the maximum batch changes.

Solution to Exercise 4 — Judging fitness for HBF

Self-study appendix B — leads for the discussion questions

Self-study appendix C — key terms

Self-study appendix D — self-assessment

• ☐ I can explain with an example that bandwidth and capacity are different constraints → the whole session ★
• ☐ In $B=$ bus width × transfer rate, I can explain in terms of power why the industry chose "wide and slow" → Exercise 2 ★
• ☐ I can estimate the stack yields of 8-, 12- and 16-high stacks almost in my head → Exercise 1
• ☐ I can say in numbers why KGD screening is essential → Exercise 1 ★
• ☐ I can explain why HBM supply does not grow quickly with money → Exercise 1
• ☐ I can name three criteria for judging which workloads suit HBF → Exercise 4
• ☐ I can explain why MoE is a promising use of HBF → Exercise 4
• ☐ I can describe the barriers holding PIM back as process (hardware) and programming model (ecosystem) → section 7

Before the next session

With these three sessions we have seen what the problem is. From Session 4 on, we turn to who makes it and how.

Sources

Introductory material chosen by the study curriculum (the "AI memory semiconductors" item)

Verified (2026-08-02, checked against several sources)

• HBM4 = JEDEC JESD270-4, published April 2025. 2,048-bit interface, 32 independent channels (2 pseudo-channels each), up to about 2 TB/s per stack, 4- to 16-high, 24Gb/32Gb dies, 64 GB per cube at 32Gb 16-high, backward compatibility with HBM3 controllers, DRFM, core voltage 1.0/1.05V and VDDQ 0.7–0.9V — checked against the JEDEC press release, TrendForce (2025-04-21) and Tom's Hardware
• HBF standardization: Sandisk–SK hynix collaboration announced 2025-08-06; OCP workstream begun 2026-02-25. Target specification 256Gb per die / 512 GB at 16-high / read about 1.6 TB/s, aiming at a footprint similar to HBM4's. First samples targeted for the second half of 2026 — checked against Sandisk and SK hynix press releases, IEEE Spectrum and Tom's Hardware. The per-die capacity and footprint wording confirmed against the original Sandisk HBF fact sheet ("256Gb per die," "closely matching")
• The three-company HBM oligopoly and SK hynix's lead by revenue (IDC figures for Q1 2026, Session 1 section 4) — verified. The order of second and third changes by generation and is not given

Verified (2026-10-08, checked against primary sources)

• HBM4 volume production: SK hynix, "SK hynix Completes World's First HBM4 Development and Readies Mass Production" (2025-09-12) — 2,048 I/O, over 10 Gbps, 1b DRAM, Advanced MR-MUF. Samsung Electronics, "Samsung Ships Industry-First Commercial HBM4 With Ultimate Performance for AI Computing" (2026-02-12) — 11.7 Gbps (up to 13 Gbps), up to 3.3 TB/s per stack, 12-high 24–36 GB, 1c DRAM and 4nm logic. Samsung Electronics, "Samsung Electronics Begins Shipment of Industry-First HBM4E Samples" (2026-05-29) — 14 Gbps (up to 16 Gbps), up to 3.6 TB/s per stack, 12-high 48 GB, "Samsung Foundry's 4nm logic base die"
• HBM4 standard: JEDEC JESD270-4 press release (2025-04-16) — 8 Gbps, 2,048 bits, up to 2 TB/s, 32 channels with 2 pseudo-channels each, VDDQ 0.7/0.75/0.8/0.9V and VDDC 1.0/1.05V, backward compatibility with HBM3 controllers, DRFM, 4-, 8-, 12- and 16-high with 24Gb and 32Gb dies (64 GB at 32Gb 16-high)
• NVIDIA Rubin: NVIDIA technical blog, "Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI" (2026-07-21) — up to 288 GB of HBM4, 12-high stacks, up to 22 TB/s (2.8 times Blackwell). The post does not give the stack count; the 8 in the text is a back-calculated estimate
• HBF: SK hynix, first HBF standard specifications (2026-08-04) — released through the OCP, 8-high and 16-high, up to 512 GB, three grades of about 0.4–3.0 TB/s, UCIe, Google and Tenstorrent participating. SK hynix, FMS 2026 roundup (2026-08-07) — HBM, DRAM, NAND (HBF), SSD hierarchy; CXL pooled-memory demonstration. Sandisk standardization collaboration press release (2025-08-06) — targets of first samples in the second half of 2026 and inference-device samples in early 2027. Sandisk HBF fact sheet (2025-07) — first generation read 1.6 TB/s, 256Gb per die, 512 GB at 16-high, CBA = CMOS directly Bonded to Array
• Stacking processes: SK hynix HBM4 press release (above) — Advanced MR-MUF. Samsung Electronics, "Samsung Develops Industry-First 36GB HBM3E 12H DRAM" (2024-02-27) — Advanced TC NCF
• SPHBM4: JEDEC JESD330-4 press release (2026-07-13) — the same DRAM dies as HBM4, 512 data signals with 4:1 serialization, mounting on organic substrates
• GDDR7: JEDEC JESD239 press release (2024-03-05) — up to 192 GB/s per device, PAM3 (3 bits in 2 cycles), independent channels 2 in GDDR6 → 4 in GDDR7. The 48 Gbps per pin is a calculated value, dividing by 32 bits
• DRAM refresh: Micron 16Gb DDR4 data sheet Rev. H (2021-08), description of the refresh function and Table 52 — 8,192 times within 64ms at 0–85°C (tREFI 7.8125μs), 32ms at 85–95°C (3.90625μs), tRFC1 160ns at 2Gb, 260ns at 4Gb, 350ns at 8Gb. The link is a copy hosted by a distributor (Digi-Key)
• CXL: CXL Consortium press release on releasing CXL 4.0 (2025-11-18) — 64 GT/s → 128 GT/s. CXL Consortium overview presentation — use of the PCIe physical and electrical interface
• Optane: Intel Form 10-Q (quarter ended 2022-07-02) — original on SEC EDGAR. It states that in Q2 2022 Intel began the wind-down of its Optane memory business and recorded an inventory impairment of USD 559 million in cost of sales in the same quarter. The next quarter's 10-Q (quarter ended 2022-10-01, original) gives the same amount as the cumulative figure for Q1–Q3 (checked 2026-10-08)

Unconfirmed / reserved (not asserted in the text)

• Measured performance of HBF — as of October 2026 there are no volume products. The 512 GB, 0.4–3.0 TB/s and 1.6 TB/s in the text are all specification ranges and targets, not verified measurements. The bandwidth of the middle grade (Grade 2) could not be confirmed from public material
• HBF sample schedule — the August 2025 target was first samples in the second half of 2026. Sandisk's August 2026 investor day material gave only the tape-out of the first HBF memory die and 2027 samples of the first inference product (checked 2026-10-08). When memory samples will ship has not been confirmed in an official announcement
• Exact stack bandwidth of HBM2E/HBM3/HBM3E — pin speeds differ by vendor and product, so the figures in the table are rough ranges. Exact values need checking against each vendor's data sheet
• When hybrid bonding will be used in volume HBM — as of October 2026 we found no manufacturer announcement that it has been used in volume HBM. Micron's stacking method was not confirmed from primary material
• Change in HBM4's command bus structure — statements listing the separation of command and data buses as an HBM4 change could not be confirmed in the JEDEC press releases (the 2024-07 preview and the 2025-04 publication), so it was left out of the list of changes in section 4. The body of the standard was not consulted
• Percentage shares by HBM maker — sources vary widely; not asserted
• The $y$ values in the section 5 yield calculation — actual makers' yields are trade secrets. The figures in the text are assumed values to show how the calculation works, not real yields
• Commercial maturity of PIM products — at the prototype and limited-deployment stage; whether they will be adopted broadly is undecided

Session 4

Fabless, Foundry and the Value Chain

Learning objectives

1. Distinguish the roles of fabless companies, foundries, IDMs, OSATs and IP vendors, and how each makes money.
2. Explain the whole chip-design pipeline by mapping it onto the software development process.
3. Set up the Poisson yield model and calculate the number of good dies per wafer for different die sizes.
4. Know which metrics you actually need to check behind the label "3nm."
5. Decide quantitatively whether to develop an ASIC.

0. Self-study guide — for readers working alone

What carries over from earlier sessions

Prerequisite check

If you have a statistics background: section 5 of this session is a rare case where the first chapter of an undergraduate probability course becomes an industry's cost structure as it stands. Exercise 3 (Poisson vs. negative binomial) reduces to one line of Jensen's inequality — see the solution in the appendix.

Reading order (110 minutes recommended)

Never skip: the table "Getting a feel for the numbers" in section 5. Looking at that table, you should be able to explain from the properties of the exponential function why yield halves when area grows 16-fold.

A common sticking point — why the GDPW formula looks the way it does

$$\text{GDPW}\approx \frac{\pi d^2/4}{A}-\frac{\pi d}{\sqrt{2A}}$$

There is nothing to fear here. Read the two terms separately.

• First term = wafer area ÷ die area. Simply "how many fit."
• Second term = a correction for edge loss. When you fill a disc with squares, some dies straddle the rim and get cut off. The second term estimates their number as wafer circumference ÷ die diagonal. If a square die has side $\sqrt{A}$, its diagonal is $\sqrt{2A}$, which gives $\pi d/\sqrt{2A}$. Take the side $\sqrt{A}$ as the denominator instead and the loss grows by a factor of $\sqrt{2}$; that one constant changes the result visibly (table below).

In short, "divide by area, then subtract what is thrown away at the rim" is all there is to it. The larger the die, the greater the weight of the second term — a big chip loses twice, once on yield and once at the rim.

This formula is an approximation too. Count directly and the difference shows. The conditions are a 300 mm wafer, square dies, zero edge exclusion and zero scribe-line width. Lay a grid of dies over the wafer and count only the dies whose four corners all fall inside the circle of radius 150 mm. We shifted the origin of the grid horizontally and vertically in steps of 1/20 of a side, counted at 400 positions, and took the mean and range. The result:

The percentages in parentheses are differences from the directly counted mean. The $\sqrt{2A}$ formula comes out 1–5% below the mean; the $\sqrt{A}$ formula, 5–19% below. The error grows with die size, and the count also depends on where the grid is placed (66 versus 71 at 800 mm²). Align one die corner with the wafer center and you get 648, 148 and 68 dies at 100, 400 and 800 mm², almost the same as the mean. A real wafer has edge exclusion and scribe lines, so even the directly counted values shrink below these. For a comparison of several approximation formulas, see de Vries (2005), doi:10.1109/TSM.2004.836656.

1. Getting the intuition — why design and manufacturing split apart

The explosion of fixed costs

The cost of making a chip comes in two lumps.

• Fixed costs (NRE, non-recurring engineering): design labor, EDA tool licenses, IP licenses, the cost of making the mask set, verification and prototypes. Paid once, regardless of volume.
• Variable costs: wafers, packaging, test. Paid per unit.

On top of these sits a far larger fixed cost: building the fab. One state-of-the-art fab costs tens of billions of dollars. To recover a fixed cost of that size, the fab has to run 24 hours a day with enormous volumes flowing through it.

Conclusion: no company can fill a fab with its own chips alone. So two paths emerged.

1. IDM (integrated device manufacturer): when a company's own product volume is large enough (as with memory, standard products made in bulk). Samsung Electronics, SK hynix, Micron, Intel.
2. The foundry–fabless division of labor: gather the designs of many companies and fill one fab with them. TSMC established this model.

Economics frame: an industry with large fixed costs and low marginal costs tends toward natural monopoly. Semiconductor manufacturing adds a learning curve (yield rises as cumulative output grows) on top, so the leader keeps gaining ground. What keeps latecomers from catching up is capital, and beyond capital, accumulated experience.

2. The players and their roles

Where the boundaries blur (2026)

• Memory makers need logic processes — the HBM4 base die is made on a logic foundry process. In April 2024, when SK hynix signed a memorandum of understanding with TSMC on HBM4, it announced that it "plans to adopt TSMC's advanced logic process for HBM4's base die" and that the two companies "will first focus on improving the performance of the base die." Collaboration between memory makers and foundries becomes a necessity.
• Foundries expand into packaging — TSMC's CoWoS and SoIC. Foundries are absorbing territory that traditionally belonged to OSATs. The reason is plain: advanced packaging decides performance, so it has to be optimized together with the front end.
• Hyperscalers become fabless companies — Google, Amazon, Meta and Microsoft design their own chips. Most of them, though, work with design partners such as Broadcom and Marvell.
• IDMs open foundries — Intel Foundry, Samsung Foundry. The question of trust (can you hand your design to a competitor?) is a structural problem.

3. The chip-design pipeline — mapped onto software development

Mapping table

Read the fourth column from top to bottom: until tape-out, the chip exists only as files. What goes to the foundry at tape-out is a layout file, and the long-standing format is the GDSII stream. SEMI has standardized OASIS (SEMI P39) as its successor format. This column is why the software analogy ends at tape-out. Below that line, fixing anything means making the object again.

Two ways it differs decisively from software

1. There are no hotfixes. If a bug turns up after tape-out, you remake the masks (a respin) and spend months and tens of millions of dollars again. That is why verification takes more than half of the total development effort. "The cost of one bug" differs from software by orders of magnitude.
2. Physics intervenes. The same RTL can perform very differently depending on the result of place and route. Timing closure (making every path meet the clock period) is the main pain of late development.

There is one more place where the "RTL = source code" analogy breaks. RTL statements are not instructions executed from top to bottom; they describe circuits that exist at the same time. Three assign statements do not add three times; they place three adders side by side, and an if is not a branch but becomes a multiplexer that picks one of two inputs. What people coming from software most often get wrong is the nonblocking assignment (<=).

always @(posedge clk) begin
  q1 <= d;
  q2 <= q1;
end

Suppose that just before the clock edge d = 1, q1 = 0 and q2 = 0. Read as top-to-bottom execution, q1 and q2 should both be 1 after the clock. In fact q1 = 1 and q2 = 0. A nonblocking assignment reads all the right-hand values first at the clock instant and then updates the left-hand sides together (the nonblocking procedural assignment provisions of IEEE Std 1364-2005; that standard has since been superseded by IEEE Std 1800, SystemVerilog). Swapping the two lines gives the same result. This code is not two instructions but a picture of a two-stage shift register made of two flip-flops in a row.

PDKs and standard cells — the abstraction layer

The foundry supplies a PDK (process design kit). It contains the design rules (DRC rules), device models (SPICE parameters) and a standard-cell library (the layouts and timing characteristics of basic blocks such as NAND gates and flip-flops).

Why this is decisive: thanks to the PDK, a fabless company can design without knowing semiconductor physics. It plays exactly the role of an API/ABI in software. And as with an API, a design made for one PDK does not carry over to another foundry as it is. The standard cells and design rules change, so everything from synthesis through place and route, timing closure and DRC/LVS has to be run again, and a new mask set has to be made. This porting cost is the real barrier to switching foundries, and it is where TSMC's ecosystem advantage comes from.

This ecosystem is a named institution. TSMC groups its partners into alliances under OIP (Open Innovation Platform®). When the Cloud Alliance was added as the fifth in October 2018, the lineup was EDA, IP, Design Center, VCA (then called Value Chain Aggregator, now Value Chain Alliance) and Cloud; in October 2022 the 3DFabric Alliance (advanced packaging and 3D IC) was formed as the sixth. Samsung Foundry does the same thing through its SAFE™ program. It bundles certified PDKs, design methodologies and IP, and runs IP, EDA, Cloud, Design Service, Multi-Die Integration (MDI) and OSAT alliances (checked October 2026). From the customer's side, choosing a foundry is not choosing one process but choosing this whole partner network.

4. The truth about "3nm" — how to read node names

What it is not

Node names once referred to a real physical dimension such as gate length. Since FinFET they have become marketing names that correspond to no physical dimension at all. Different companies' "3nm" processes differ in density and performance, and even across generations at the same company the name does not track the performance gain directly.

Then what should you look at?

What matters more for AI accelerators: AI chip performance depends more on memory bandwidth and packaging than on the node. Attaching one more HBM stack can affect performance more than being one node generation ahead.

The 2nm class as of 2026

Status as rechecked on October 8, 2026 against each company's official announcements.

The contrast to watch: Intel chose the aggressive strategy of introducing GAA and backside power delivery at the same time; TSMC chose the conservative one of stabilizing GAA first and pushing backside power delivery to the next node. This is not a difference in technical superiority but a difference in risk-budgeting strategy. It makes a good discussion topic.

5. The economics of yield — this book's second key tool

The Poisson model

Assume that defects land on the wafer at random (spatially independent and uniform). If the average density of killer defects per unit area is $D_0$ [defects/cm²] and the die area is $A$ [cm²], the number of defects landing on one die follows a Poisson distribution with mean $\lambda =A{D}_0$. The probability of no defect at all, that is, the yield, is

$$Y=P(X=0)=e^{-A{D}_0}$$

For statistics majors: this is just the $P(X=0)$ of a Poisson distribution. One of the pleasures of this field is that semiconductor yield theory starts from the first chapter of undergraduate probability.

Where the formula comes from — cutting the die into cells. Divide the die into $n$ small cells of equal area, and let the probability of a defect landing in each cell be $p=\lambda /n$. If the cells are independent, the probability that the die is clean is the probability that every cell is clean, and as you cut the cells ever finer this becomes the high-school limit ${lim}_{n\to \infty }(1+x/n{)}^n=e^x$ with $x=-\lambda$.

$$P(\text{0 defects})={(1-\frac{\lambda }{n})}^n⟶e^{-\lambda }(n\to \infty )$$

Calculating, for $\lambda =0.8$ you get 0.4344 at $n=10$, 0.4479 at $n=100$ and 0.4492 at $n=\mathrm{1,000}$, while the limit $e^{-0.8}$ is 0.4493. Splitting into just ten cells already reaches 97% of the answer.

Getting a feel for the numbers

With $D_0=0.1$ defects/cm² (an illustrative value at the level of a mature process):

When area grows 16-fold, yield falls below half. And because a large die also fits fewer times on a wafer, the number of good dies falls much more steeply.

Good dies per wafer

The area of a 300mm wafer is about $\pi \times {15}^2=706$ cm². There are several approximation formulas for gross die per wafer that account for edge loss; a common one is

$$\text{GDPW}\approx \frac{\pi d^2/4}{A}-\frac{\pi d}{\sqrt{2A}}$$

($d$ = wafer diameter, $A$ = die area. The second term is the edge-loss correction: the wafer circumference divided by the die diagonal $\sqrt{2A}$. The comparison showing that this formula comes out 1–5% below the directly counted number for 100–800 mm² dies is in "A common sticking point" in section 0.)

Good dies = GDPW × $Y$, and cost per die = wafer price / number of good dies.

Exercise: with the price of one wafer as $W$, find the ratio of cost per die for die areas of 100 mm² and 800 mm². The area ratio is 8, but the cost ratio comes out much larger.

Why the Poisson model is wrong — defect clustering

Real defects do not scatter independently; they cluster (particle contamination, tool faults, wafer-edge effects). With clustering, the Poisson model underestimates yield, because when defects crowd onto a few dies, the remaining dies stay clean.

Alternative models

In the negative binomial model, $\alpha$ is the clustering parameter: as $\alpha \to \infty$ it converges to Poisson, and the smaller $\alpha$, the heavier the clustering. That a gamma–Poisson mixture becomes a negative binomial is a familiar result in statistics, and the semiconductor yield model has exactly that structure. If the defect density $D$ follows a gamma distribution with mean $D_0$, its variance is $D_0^2/\alpha$. $\alpha =1$ means the standard deviation of the defect density equals its mean (the integral is in the solution to Exercise 3). The memory yield model IBM published in 1980 was also a mixture model that shook the Poisson mean (Stapper et al., 1980).

Going deeper: this section is where someone trained in statistics can most comfortably dig deeper. In real fabs, spatial statistics and machine learning are applied to wafer-map data (which dies at which positions failed) to classify defect patterns and trace them back to the tool that caused them. This connects to the inspection and metrology section of Session 5.

$A$ is not the whole die area — critical area. Not every defect that lands causes a failure. A particle that falls between two wires does not short them unless it is big enough to touch both, and one that lands where there is no circuit does nothing. So strictly, the $A$ in $\lambda =A{D}_0$ is not the die area but the critical area: the area of the places where a defect of that size would break the circuit. On the same die, the larger the defect, the larger the critical area, and densely wired blocks have more of it. IBM's memory yield model (Stapper et al., 1980) described the product by this critical area and the process by the defect density, separately. When this book calls $D_0$ the "killer defect density" and uses the die area as it is, that is a simplification that absorbs the effect of critical area into the density.

What yield explains

1. Why chiplets — making four 200 mm² dies (82% each) separately, testing them, and binding only the good ones (KGD) is far better than one 800 mm² die (45% yield). Bind the four without sorting and the finished-product yield is $(e^{-0.2}{)}^4=e^{-0.8}\approx 45\%$, the same as the single die (the multiplicative yield of Session 3). The gain comes not from cutting the die smaller but from throwing out the bad ones before assembly so that only good silicon is used. The cost of joining them and the assembly yield are added, however. That profit-and-loss calculation is the core of Session 6.
2. Why foundries sell by the wafer — it is a question of who bears the yield risk. Under a per-wafer contract the fabless company bears it; under a per-good-die contract the foundry does. The newer the node, the more the foundry prefers per-wafer terms.
3. Why new nodes are expensive — early on, $D_0$ is high. Over time it comes down along the learning curve (yield learning). This is the structural advantage of the front-runner.
4. Why binning is done — binning comes in two kinds.
   • Defect binning. Some defects leave the chip usable if the affected core is disabled. Selling an 8-core design as a 6-core product is a strategy for recovering yield. Defects can also be repaired. Memory keeps spare rows and columns and swaps the defective lines for spares (repair), and chips that work only in part are sometimes sold separately. IBM, reporting the yield trend of its 64K-bit RAM, also presented the yield gains it obtained from redundant circuits and partially good products (Stapper et al., 1982). The redundant circuits of HBM in Session 3 work on the same principle.
   • Speed and power binning. Even among defect-free dies, process variation makes the maximum operating frequency and power differ. So each die is tested for its maximum frequency and graded. This is a matter of parametric yield, which $e^{-A{D}_0}$, a formula that counts defects, cannot explain. Where defect binning deals with "how many are alive," speed binning deals with "how fast the living ones are."

6. ASIC, FPGA or general-purpose processor — a quantitative choice

Break-even (a refinement of the formula from Session 2)

$$N^{*}=\frac{C_{\text{NRE}}}{(c_G+o_G)-(c_A+o_A)}$$

• $c$: unit acquisition cost; $o$: present value of lifetime operating cost (mainly electricity)
• As in Session 2, assume one ASIC replaces the work of one GPU 1:1. If throughput differs, $N$ has to be converted into the number of units that handle the same demand
• $c_A$ is calculated from the cost per die in section 5 above + packaging + test

Trade-offs on five axes

The trap on the time axis

Even with enough volume to clear $N^{*}$, it is no use if the period over which that chip handles that volume is short. If model architectures change substantially every two years and chip development takes two to three years, the computation pattern the chip targeted must still dominate when it is finished.

Why hyperscalers can take this risk

1. Their volume $N$ is overwhelming.
2. They control their own workloads. Being able to design the model to fit the chip is decisive.
3. The total savings on electricity are large (at data center scale). Per unit, it is mainly the difference in acquisition price that decides the break-even count, and the difference in electricity cost amounts to only a few percent to a dozen-odd percent (solution to Exercise 4 in Session 2, solution to Exercise 5 in this session); but run tens of thousands of units for several years and that share also becomes a large sum.
4. There is option value in bargaining power with vendors.
5. They have the capital to absorb a failure.

Why most companies do not: none of the five holds for them.

7. Common misconceptions

Misconception 1: "TSMC also designs chips" → The core of the pure-play foundry model is the trust that it does not compete with its customers. It offers IP blocks and packaging services but does not design finished chips. This neutrality is a large part of TSMC's success.

Misconception 2: "A more advanced node makes a faster chip" → For AI accelerators, memory and packaging can matter more. And the node name itself is not a comparable metric.

Misconception 3: "Yield converges to 100% as a process matures" → Inside the Poisson formula, that is right. Whatever the area, $e^{-A{D}_0}\to 1$ as $D_0\to 0$. The trouble is that $D_0$ never goes all the way to 0. At the same $D_0$, yield falls exponentially with die size, so even on a mature process a large die stays at a lower yield than a small one (an 800 mm² die is at 67.0% even with $D_0=0.05$). On top of that, parametric losses that are not counted as defects (speed and power binning in section 5) remain outside the formula. And a new node always starts again from low yield.

Misconception 4: "A fabless company doesn't need to know manufacturing" → The PDK abstracts it away, but high-performance design has to take process characteristics (variability, electromigration, heat) deeply into account. That is why DFM (design for manufacturability) exists as a field of its own.

Misconception 5: "An OSAT is a low-value-added subcontractor" → Its standing has risen sharply now that advanced packaging decides performance. The leading edge, however, is being absorbed by foundries, so the OSAT's position is in flux.

Misconception 6: "EDA is just CAD software" → Place and route and timing optimization are large-scale combinatorial optimization problems, and reinforcement learning and graph neural networks are now actually being applied to them. This is the most direct point of entry for people trained in CS and AI.

8. Quantitative exercises

Exercise 1 — Yield curves For each of $D_0=0.05,0.1,0.2$, plot the yield curve over die areas of 50–800 mm². From what area does yield drop sharply?

Exercise 2 — Cost per die With the price of a 300mm wafer as $W$, find the cost per die for die areas of 100 / 400 / 800 mm² as multiples of $W$. Apply both the GDPW approximation and the yield.

Exercise 3 — Poisson vs. negative binomial For the same $A{D}_0$, compare the yields of the Poisson model and the negative binomial ($\alpha =1,2,5$). Confirm that the heavier the clustering, the more the actual yield exceeds the Poisson prediction, and explain why in probabilistic terms.

Exercise 4 — Chiplet profit and loss (preview of Session 6) Compare a single 800 mm² die with four 200 mm² chiplets. On the chiplet side, include (a) a 10% area overhead for the interface circuits between chiplets, (b) an assembly yield of 95% and (c) the added cost of the interposer and assembly. Take (c) as the cost loaded onto one finished product (including the share of failed assembly attempts). Which side wins? Is there a point where changing $D_0$ reverses the conclusion?

Exercise 5 — The ASIC decision Assumptions: NRE USD 150 million, ASIC unit cost USD 4,000, commercial GPU unit cost USD 30,000, three-year electricity cost USD 2,000 for the ASIC and USD 6,000 for the GPU. Find $N^{*}$, and discuss which organizations could realistically reach that volume.

9. Discussion questions

1. Can the foundry–fabless division of labor be reversed? Which direction does each of Intel's IDM 2.0 strategy and hyperscalers' vertical integration point in?
2. Between the risk-budgeting strategies of Intel (GAA and backside power at once) and TSMC (GAA first, backside power later), which is rational? Under what conditions is each right?
3. Yield data are trade secrets. What observable proxies could be used to estimate yield from outside? (Prices, lead times, product lineups, binning structure, etc.)
4. Why does the three-firm EDA oligopoly not break? Could open-source EDA (OpenROAD and others) and open PDKs change that?
5. Now that node names have lost their physical meaning, why does the industry keep using that naming scheme? Is there an alternative?

10. Further study

• Standard VLSI design textbook: Weste & Harris, CMOS VLSI Design: A Circuits and Systems Perspective — sets out models of yield, area and power
• Yield theory: Stapper's papers on the IBM yield model — Stapper, McLaren, Dreckmann, "Yield Model for Productivity Optimization of VLSI Memory Chips with Redundancy and Partially Good Product," IBM J. Res. Dev. 24(3):398–409 (1980), doi:10.1147/rd.243.0398 · Stapper et al., "Evolution and Accomplishments of VLSI Yield Management at IBM," IBM J. Res. Dev. 26(5):532–545 (1982), doi:10.1147/rd.265.0532. For a comparison of several yield models, Cunningham, "The Use and Evaluation of Yield Models in Integrated Circuit Manufacturing," IEEE Trans. Semiconductor Manufacturing 3(2):60–71 (1990), doi:10.1109/66.53188
• GDPW approximations: de Vries, "Investigation of Gross Die Per Wafer Formulas," IEEE Trans. Semiconductor Manufacturing 18(1):136–139 (2005), doi:10.1109/TSM.2004.836656 — worth reading after you have worked out the table in section 0 yourself
• Try it yourself: Tiny Tapeout — a project that lets you put your own design on a real chip for a small fee. The best way for someone with a CS background to feel the tape-out pipeline firsthand
• Open toolchain: OpenLane / OpenROAD + SkyWater 130nm open PDK — lets you run from RTL to GDS for free
• AI × EDA: Mirhoseini et al., "A graph placement methodology for fast chip design," Nature 594 (2021), doi:10.1038/s41586-021-03544-w — placement based on reinforcement learning. The paper's reproducibility was disputed, so read the later rebuttals and replies along with it
• Industry structure: IR materials and technology-symposium presentations from TSMC, Intel and Samsung — TSMC quarterly results · Intel press releases

Self-study appendix A — solutions to the quantitative exercises

Solutions to the exercises in section 8. All areas are converted to cm² before being plugged in (100 mm² = 1 cm²).

Solution to Exercise 1 — Yield curves

Solution to Exercise 2 — Cost per die

Solution to Exercise 3 — Poisson vs. negative binomial