The weights and the KV cache share one accelerator's memory. The capacity left over sets the batch size, and the batch size sets the arithmetic intensity. Change the sequence length and the precision and watch how a capacity limit ends up holding down bandwidth efficiency as well.
KV/seq = 2 · L · S · d · bkv · Bsz = ⌊(M − P·b − reserve) / KV⌋ · tok/s ≲ B / (P·b) × Bsz
How one accelerator's memory is divided
WeightsReserve (activations, fragmentation)KV cache (one cell per sequence)Unused
Chip
Weight precision
KV cache precision
Attention (number of KV heads)
Weights
KV / sequence
Capacity available for KV
Max batch size
Throughput ceiling
Compute utilization ≈ Bsz/I*
How fast the batch size shrinks as sequences grow
Max batch size for three weight precisions at the currently chosen chip, KV precision, attention and reserve ratio. The dashed line is the batch size needed to reach the roof (Bsz ≈ I*).
Check against the solution tables ()
H100 80GB, S = 4,096, KV fp16, reserve 0%. The "Solution" column holds the rounded values from the solution tables of Session 2, Exercise 3 and Session 3, Exercise 3.
Item
Computed
Solution
What this calculation leaves out
Utilization is counted as Bsz / I*(BF16). If computation also runs at the weight precision, the actual arithmetic intensity 2Bsz/b and that precision's ridge point (2/b)·I* grow together, so their ratio equals this value (table in Session 2, solution to Exercise 3). If only the weights are read at low precision and computation runs in BF16, arithmetic intensity rises by a factor of 2/b and utilization comes out that much higher (this is the case the precision buttons on the roofline bench show).
The throughput ceiling counts only the bandwidth side. Once the batch size passes the ridge point the workload becomes compute-bound, so this formula overstates the ceiling. The page flags that case separately.
Traffic for reading the KV cache is left out. With long contexts the KV cache must also be read at every step, so real speed is lower.
Every sequence is assumed to fill up to the maximum length S. With schemes like PagedAttention that allocate only what is used, the average length is what counts.
Capacity is counted in decimal GB (109 bytes), as in the text.