Notes · Weekly roundup
Week of October 7: shipping weight diffs instead of checkpoints
A trillion-parameter refit drops to 150 seconds by sending only the weights that changed, vLLM adds two ways to keep an engine warm across restarts, Aleph Alpha ships Kolibri-1 with FP8 weights, and a 975B model runs across eleven PCs.
Four items this week, all about where weights sit and how they move: across regions during training, between engine restarts, at 8 bits on a single GPU, and spread over eleven desktop-class machines.
Refitting a 1T model by sending the diff
In reinforcement learning, the trainer has to push fresh weights to the inference fleet after every step. The NeMo-DCR paper, whose code is open-sourced in NVIDIA NeMo RL, measures that a full 1T checkpoint takes 87.5 minutes to move between two AWS regions, while in BF16 training “about 1% of weights change their stored values per step”. Sending a bit-exact delta over a relay tree instead brings a 1T refit at a 3% change rate down to 150 seconds. For anyone who stores checkpoints, the point is that consecutive versions of a model are mostly identical bytes, and the transfer only needs to carry the ones that differ.
Source: NeMo-DCR (arXiv 2610.08430)
vLLM 0.31 keeps weights in memory across restarts
vLLM’s new release adds vllm preload, a daemon that “keeps post-quantized weights resident in GPU memory across engine restarts”, and an experimental vllm snapshot create/restore that uses CRIU to restore “a fully initialized TP1 engine”, one that runs without tensor parallelism. The notes file both under “Fast restart”: the daemon spares a restarted engine the weight load, and the snapshot spares it the initialization as well. The release notes give no restart timings, so the saving is still something to measure. The same release makes an NVFP4-compressed KV cache the default for DeepSeek-V4.1-Flash on Blackwell (SM100) GPUs.
Source: vLLM v0.31.0 release notes
Kolibri-1 ships its weights in FP8
Aleph Alpha’s Kolibri-1 is an Apache-2.0 MoE with 78.1 billion parameters, 3.46 billion of them active per token. Of the 78,103,074,560 parameters in its files, 77,398,016,000 are stored as FP8 in 128×128 blocks; the embeddings, LM head, norms and router stay in BF16. The card puts the footprint at about 78 GB, enough for one H200 or B200, and it was evaluated with an FP8 KV cache. Its config makes that cache small: only 10 of the 50 layers use full attention, each with 4 KV heads of dimension 128, so a token costs 10,240 bytes in FP8 (my arithmetic from the config). The other 40 layers see a 512-token sliding window, so their cache does not grow with the prompt. At the card’s maximum context of 1,048,576 tokens the full-attention cache is ≈10.7 GB, half what BF16 would need.
Source: Kolibri-1 model card and config.json
A 975B MoE resident on eleven PCs
The Cascadia paper keeps Inkling, a 975B-total, 41B-active model, entirely in memory across eleven Intel AI PCs with 64 GB each, integrated graphics and gigabit Ethernet. Each machine holds six consecutive decoder layers. The cluster reaches 60.29 aggregate decode tokens per second at 88 streams, with a median first-token time of 6.05 seconds at fifteen streams. On prompts of 1k to 64k tokens, the authors find the bound is a single-threaded CPU attention loop rather than memory. It is a concrete case of MoE serving where capacity comes from many small memory pools and only the active experts do work.
Source: Cascadia (arXiv 2610.07219)
The latest Signal & Weight part covers how bits per weight set the GPU count: Quantization: the dial that decides how many GPUs you need.
More in Weekly notes. For the long explainers, start with Signal & Weight.