The break-even bandwidth: when loading a KV cache beats recomputing it

In late 2025, VAST Data and CoreWeave sent a 127,188-token prompt to Llama 3.1 405B on eight H100s. Computed from scratch, the first token took 62 seconds. With the prompt’s KV cache loaded from shared storage, it took 3. The post calls that a 20× speedup.

The same post holds enough to work out a number it doesn’t print: the slowest link at which loading would still have won. I get 1.06 GB per second, or 8.5 Gbit/s. A single 10 GbE link running flat out would clear it.

That number is the break-even bandwidth. This post derives it, checks it against three published tests, and shows what it does and doesn’t settle.

What gets cached

For every token of a prompt, each attention layer stores a key and a value for each of its KV heads. The size per token is

$$2 \times \text{layers} \times \text{KV heads} \times \text{head dimension} \times \text{bytes per number}$$

Llama 3.1 405B has 126 layers, 8 KV heads and a head dimension of 128. At 16 bits that is 516,096 bytes per token, so the 127,188-token prompt carries a 65.6 GB cache. VAST’s post gives the same figures in binary units, 504 KiB and about 61 GB.

Building those 65.6 GB is the prefill, and it’s the 62 seconds. A model that has seen the prompt before can skip that work if the cache was kept somewhere. Only prefill benefits. Each token the model generates afterwards is new, so nothing stored can stand in for it, and the per-token speed in these tests doesn’t change.

Two ways to get the same bytes

Say the prompt has $N$ tokens, the GPUs prefill $r$ tokens per second, each token’s cache is $b$ bytes, and the link delivers $B$ bytes per second into GPU memory.

$$t_\text{recompute} = \frac{N}{r} \qquad t_\text{load} = \frac{N\,b}{B}$$

Loading wins when $B > r \times b$. The prompt length cancels. To a first approximation, a link is either fast enough for a model on given hardware or it isn’t, whatever the prompt.

Three published tests give both inputs:

Model and hardware	Prefill	Cache per token	Break-even
Llama 3.1 405B, 8 H100s	2,051 tokens/s	516 KB	1.06 GB/s
Llama 3.1 70B, DGX H100	4,380 tokens/s	328 KB	1.44 GB/s
Qwen3-32B, 1 H100	≈11,900 tokens/s	262 KB	≈3.1 GB/s

The prefill speeds are each test’s uncached time to first token divided into its prompt length, so they include a little queueing and one decode step. The Qwen row is approximate because its source says “over 11 seconds”.

The bar falls as the model grows. From Qwen3-32B to Llama 3.1 405B the cache per token doubles, and prefill gets 5.8 times slower even with eight GPUs in place of one. Large models on long prompts are where loading wins most easily, and they are the models these demonstrations use.

The multiplier is two bandwidths divided

Once a link is above break-even, the speedup is $t_\text{recompute} / t_\text{load}$, which reduces to $B / (r \times b)$: the link’s speed over the break-even.

VAST reports that its test sustained 181 Gbit/s over two 100 Gbit/s links, which is 22.6 GB/s. Divided by 1.06, that predicts 21×, and a load time of 2.9 seconds. The post reports 20× and 3 seconds.

The storage system doesn’t appear in that formula. It has to be able to feed the link, and past that point a faster one changes nothing. VAST’s post says as much: its offload was “not storage-limited, but rather they are network-limited”.

The same arithmetic is a check on other claims. Qwen3-32B’s 131K-token cache is 34.4 GB, and loading it in the reported 1.5 seconds needs 22.9 GB/s, which is plausible on the hardware described. WEKA’s Llama 3.1 70B result, 105,000 tokens in 0.58 seconds, needs 59 GB/s for a 34.4 GB cache. One 400 Gbit/s InfiniBand link carries 50. The test either used more than one link or moved a smaller cache, and the write-up I read doesn’t say which.

TCP is fast enough, and TCP is far too slow

The llm-d project measured three ways of moving data between GPUs over the same 100 Gbit/s network card:

Transport	Delivered	Against the three break-evens
RoCE (RDMA over Ethernet)	12.0 GB/s	wins all three
TCP, UCCL backend	4.9 GB/s	wins all three
TCP, UCX backend	0.55 GB/s	loses all three

Two TCP stacks on one card differ by a factor of nine, and the break-even runs between them. That explains how VAST could report its 20× result over NFS on TCP in 2025 and write in 2026 that on “a standard TCP NFS share” the load-faster-than-recompute condition fails. Both statements can hold. What matters is the bandwidth that reaches GPU memory, and “TCP” covers a wide range of it.

Where the cache lives

The map at the top of the interactive uses NVIDIA’s names for the places a cache can sit. G1 is GPU memory, where the cache has to be while the model reads it. G2 is the server’s own DRAM and G3 its local SSD. G4 is shared storage that every server in the cluster can reach. NVIDIA has since added G3.5, flash shared within one pod of servers.

A request can take two routes across that map. On a miss, the prefill GPUs compute the cache and hand it to the GPUs that will generate, a transfer that happens once per request. On a hit, the cache comes up from one of the lower tiers and prefill is skipped. When a conversation pauses, its cache moves down a tier to make room, and how far down it goes decides who can reuse it. A cache in G2 or G3 is only useful if the next request lands on the same server. One in G4 can be picked up anywhere.

The break-even applies to whichever link the hit route uses.

The hit rate sets the average

A break-even says what happens on a hit. A workload is a mix of hits and misses, and a miss can cost more than it did before, because the new cache has to be written out.

VAST and Lablup published a test that shows both sides. Ten agent contexts of about 140K tokens ran five turns each on Mistral Medium 3.5:

	No offload	With offload
First turn of a context (miss)	≈22.3 s	≈26.3 s
Later turns (hit)	≈22.0 s	≈6.6 s
Average time to first token	22.1 s	10.6 s
Time per output token	14.5 ms	14.5 ms

The first turn got 4 seconds slower. The average still halved, because four turns in five were hits. The whole run finished 1.96 times sooner.

With hit rate $h$ and a miss cost $p$, the average is

$$h \times t_\text{load} + (1-h) \times (t_\text{recompute} + p)$$

and offloading pays for itself once $h$ exceeds $p / (p + t_\text{recompute} - t_\text{load})$. With this test’s numbers that is 4 / (4 + 22.0 − 6.6), a hit rate of 21%.

For comparison, DeepSeek reported that over one day in production, 342 billion of its 608 billion input tokens hit a KV cache on disk: 56.3%.

The test had one more cost that a time-to-first-token figure doesn’t show. Each GPU set aside 24 GiB of its memory as a transfer buffer, memory that would otherwise hold live caches.

Newer models move the problem

Everything above uses Llama-era attention, where the cache is large. The models in Inside Open Models store far less per token:

Model	Cache per token	A 128K-token conversation
Llama 3.1 405B	516 KB	67.6 GB
Qwen3.8-Max	94 KB	12.9 GB
Kimi K3	27.6 KB	4.1 GB
DeepSeek V4 Pro	7.9 KB	1.05 GB

The per-conversation sizes include each model’s fixed state, as Part 1 did for Qwen. DeepSeek V4 Pro keeps 65 times fewer bytes per token than Llama 3.1 405B. A full million-token conversation on it is 8.3 GB.

None of these models has a published prefill time I could find, so I can’t give their break-evens. The formula does set a bound. For DeepSeek V4 Pro’s break-even to reach even 1 GB/s, it would have to prefill 126,000 tokens per second. Short of that, any modern link clears the bar.

For these models the question shifts from how fast the link is to the hit rate, how long a paused conversation is worth keeping, and who is allowed to reuse it. Capacity shrinks with the cache as well. A thousand parked 128K-token conversations take 67.6 TB on Llama 3.1 405B and about 1 TB on DeepSeek V4 Pro.

What this leaves out

The formula treats prefill as a constant speed. Real prefill slows as the prompt grows, so long prompts favour loading a little more than the formula says. It also ignores the fixed cost of a load, the lookup and the connection, which favours recomputing on short prompts.

Each prefill speed here comes from a single run reported by the company selling the system, on the prompt length it chose. I’ve used them because they are the only public figures that give both inputs.

Cache sizes are for 16-bit, uncompressed caches. Compression lowers the break-even in proportion. Part 3 covered TurboQuant, which reports more than 5× on the cache.

Sharing a cache between users is where the largest savings are and also a way to leak prompts. A 2025 audit found evidence of cache sharing across users at seven API providers, detectable from response times alone.

Where this goes next

The calculator takes any of 13 models, your own prefill speed and your own link. Part 6 of Signal & Weight will come back to the cache from the model’s side: why Qwen’s is 94 KB per token and DeepSeek’s is 8.

Sources: VAST Data and CoreWeave, NVIDIA Dynamo and VAST (2025), for the Llama 3.1 405B test, the 504 KiB and 61 GB figures and the 181 Gbit/s throughput; the test used AWQ 4-bit weights, modified pre-release software and one prompt. VAST Data and Lablup, KV cache offload benchmark (2026), for the Mistral Medium 3.5 table, the 24 GiB buffer and the TCP NFS statement. VAST Data, A practical guide to right-sizing KV cache (2026), for the point that reuse only helps prefill and for the tier names. Qwen3-32B: World Wide Technology, KV cache: vendors and academia, reporting VAST’s figures. Llama 3.1 70B: NAND Research, WEKA Augmented Memory Grid, reporting WEKA’s test. Transports: llm-d, Networking for distributed inference. Hit rate: DeepSeek, V3/R1 inference system overview (2025). Cache sharing: Gu et al., Auditing Prompt Caching in Language Model APIs (ICML 2025). Layer and head counts from the Llama 3.1 70B, Llama 3.1 405B and Qwen3-32B configs; the other models’ cache sizes are from Inside Open Models. Break-even bandwidths, prefill speeds, predicted load times and the 21% hit rate are my arithmetic on those inputs.