The Thinking LayerKV cacheload or recompute

Load or recompute?

A cached prompt can be loaded from memory or storage, or computed again by the GPUs. Loading wins when the link moves bytes faster than prefill creates them.

DeepSeek reported that 56% of its input tokens hit a cache on disk over one day in production.
Writing a new cache out costs time. One test measured about 4 s on a 140K-token first turn.

Where the cache lives

A request reaches the router. On a miss the prefill GPUs compute the cache and hand it to the decode GPUs. On a hit the cache is loaded from a storage tier and prefill is skipped. When a session pauses, its cache is parked in a lower tier.
  • Miss: prefill GPUs compute the cache
  • Hit: the cache loads from a tier, prefill is skipped
  • Park: the session pauses and its cache moves down

Break-even bandwidth

Where each link fallsGB per second, log scale · click to set

Links on a bandwidth axis, with the break-even line.
This prompt's cache
Time to first token, averaged
Hit rate needed to come out ahead
1,000 parked sessions

Model: recompute time = tokens ÷ prefill speed, load time = cache bytes ÷ link speed. Real prefill slows as prompts grow, and a load also pays a fixed cost for lookup and connection, so short prompts favour recomputing more than this shows. Prefill speeds marked measured are single vendor-reported runs, taken as time to first token. Cache sizes are 16-bit, uncompressed. Explained in The break-even bandwidth.