This page lets you move one inequality from Session 2, Section 6 by hand. Change the batch size and precision, and see for yourself which side of the roof the workload lands on and what percentage of the chip it uses.
Pattainable(I) = min( Ppeak , B × I ) · I* = Ppeak / B
Plotting a workload on the roof
Chip (BF16 dense)
Phase
Weight precision
Arithmetic intensity I
Ridge point I*
Attainable performance
Compute-unit utilization ceiling
Verdict
Upper bound on decode speed
A rough estimate that restates the same inequality in tokens. The answer is the smaller of the bandwidth-side and compute-side bounds. Chip, precision and batch size are taken from the settings above.
The chip-count slider sets how many chips to add beyond the capacity minimum. This calculation assumes the N chips split the weights and read them in parallel, so it adds bandwidth N times over; each extra chip raises the bandwidth-side bound in proportion (405B fp16 at batch 1: 11 chips 45 → 12 chips 50 tok/s). In practice, inter-chip communication keeps it from rising that much (see "What this chart does not say" below). At the minimum chip count, try 70B and 405B and check that, whatever the model size and precision, the batch-1 bound sits just above one chip's capacity ÷ bandwidth (about 42 tok/s for the H100). The η knob is there so you can enter the "60–80% bandwidth utilization" from the text.
Check against the solution tables ()
This compares the figure's calculations with the table and text of Session 2, Section 6 and the solutions to Exercises 1 and 2 (Exercise 2 uses H100 SXM, batch 1, η 100%). The "Solution" column holds the rounded values from the solution tables.
Item
Computed
Solution
What this chart does not say
The roofline gives an upper bound that a long-running kernel cannot exceed. It is not a model that predicts actual speed. The limits listed in Session 2, Appendix B are repeated here.
Time to first token. Only the steady state is covered.
The cache hierarchy. HBM, L2 and SRAM are collapsed into a single bandwidth B.
Inter-node communication. The tensor-parallel communication cost that comes with adding chips is left out.
Kernel launch and synchronization overhead, and clock throttling under power limits.
KV cache traffic. Decode is assumed to read only the weights. With long contexts, KV cache reads come on top, and the real arithmetic intensity is lower (Exercise 3).
Peak of low-precision compute units. Even if you pick INT8 or INT4, Ppeak stays at the BF16 dense value. This is the case where only the weights are read at low precision and the math runs in BF16. If the hardware supports doing the math at the same precision too, peak and arithmetic intensity both rise, and utilization ends up close to the fp16 case. The utilization column in Session 2, solution to Exercise 3 (I ≈ Bsz) is an estimate of that kind and comes out lower than this page.