Notes · Weekly roundup
Week of October 9: loading a KV cache from disk instead of recomputing it
A 50-million-token test reads saved KV blocks back from NVMe faster than a GPU can rebuild them, a placement scheme stretches the life of high-bandwidth flash, two MoE papers cut the cost of moving tokens between experts, and a forgetting study points at one setting on the output layer.
Most of this week’s papers are about the bytes that sit next to the weights: the KV cache, and the tokens that travel between GPUs on their way to an expert. One is about which weights change when a model is trained further.
Reading the KV cache back from disk
The galahad-kv paper saves the KV state of each block of about 16,000 tokens to encrypted local NVMe and loads it back byte-exact when the text comes up again. Served through vLLM on one H100 with Gemma 4 12B and 31B over 50,000,000 tokens, loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy. The authors are clear about the limits: one block is loaded at a time, so this is reuse of stored state and the attention window stays the same size, and the store takes terabytes of disk. It is a measured case of the trade in the break-even post: a cache load beats a recompute once storage bandwidth clears a threshold.
Source: galahad-kv (arXiv 2610.10845)
Placing agent KV cache by how long it lives
High-bandwidth flash offers “an order of magnitude greater capacity at HBM-class read bandwidth”, but each cell survives only a limited number of writes. Lachesis sits between the agent harness and the serving engine and puts each KV segment in HBM or flash according to its lifetime, so more of an agent run’s writes land in HBM and fewer reach the flash. In trace-driven simulation that extends flash lifetime by 1.19-3.13x over filling HBM first, reaching 3.3-12.2 device-years.
Source: Lachesis (arXiv 2610.08378)
Experts that are picked together
On 8-GPU MI300X nodes, the all-to-all that sends tokens to their experts can take 45% of a training step at EP32 with top-2 routing and 60% with top-6. The Expert Coupling paper finds that routers settle into correlated choices early in pretraining: at top-2, 0.8% of the expert pairs in a layer are selected together by 42% of tokens. Putting coupled experts on the same GPU, plus shuffling tokens, cuts all-to-all time by 1.16-2.63x and step time by up to 1.41x, without changing any routing decision or expert weight.
Source: Expert Coupling (arXiv 2610.09372)
MoE inference over PCIe through host memory
Consumer GPUs have no peer-to-peer link, so every token routed to an expert on another card crosses the PCIe bus via host RAM. CoMoE builds on that and makes the host the routing hub: a token bound for several GPUs is written to host memory once, and results are combined per token in host staging buffers instead of waiting on a global sync. On RTX 5090s it improves throughput by up to 1.46x and approaches NVLink A800s at 23.4% of the hardware cost.
Source: CoMoE (arXiv 2610.09424)
Forgetting sits in the output embeddings
When a model keeps training on new data, tokens that the new corpus rarely uses still receive persistent one-sided gradients, and Adam’s normalization scales those into full-sized updates of their output embeddings. Raising Adam’s epsilon for the output projection alone removes 39.4% to 67.9% of forgetting across seven stable configurations, in settings from 160M to 12B parameters. For anyone diffing checkpoints, it says where much of the unwanted change lands: in the LM-head rows of tokens the new data rarely uses.
Source: A Deafening Silence (arXiv 2610.09835)
The latest Signal & Weight part covers how bits per weight set the GPU count: Quantization: the dial that decides how many GPUs you need. The arithmetic behind the first item is in the break-even bandwidth post.
More in Weekly notes. For the long explainers, start with Signal & Weight.