Signal & Weight · Part 2 of 8
What weights actually store
Qwen3.8-Max is 2.50 TB of weights, and not one sentence in it can be looked up. Here's what is in there instead, and why "compression" describes it better than "database".
In Part 1 I followed one token through Qwen3.8-Max and came out with two numbers: about 2.45 trillion parameters, and 2.50 TB of files to hold them in FP8.
It’s tempting to picture that as a very large database. The facts are in there somewhere, and the prompt goes and finds them.
That picture is wrong. And it’s wrong in a way that decides how these files can be shrunk, split and loaded, which is the rest of this series. So before moving any bytes, I want to know what the bytes are.
There’s no sentence in there
A weight is one number. In Qwen’s FP8 release, 98% of them take one byte each.
The model is 2,446,182,725,504 of them, grouped into matrices. That’s all. No strings, no index, no row keyed by topic. “Why is the sky blue?” has no answer stored at any address.
What the matrices do is multiply. The prompt becomes a 14 × 8,192 grid of numbers, and every layer multiplies that grid by its weights and adds the result back. The answer is computed, one token at a time. Nothing is retrieved.
What training squeezes in
So what are 2.45 trillion numbers for? One job: predicting the next token.
And prediction is compression. Here’s the arithmetic. Writing down a token nobody could predict, out of 248,320 possible, takes 17.9 bits. If the model gave that token a 50% chance, it takes 1 bit.
The better the prediction, the shorter the file.
A DeepMind paper took this literally and used a language model as a file compressor. Chinchilla 70B squeezed 1 GB of Wikipedia to 8.3% of its size, where gzip manages 32.3%. Trained primarily on text, it also beat PNG on images (48.0% against 58.5%) and FLAC on audio (21.0% against 30.9%).
Those rates leave out the size of the model itself. Count it, and the paper says a model like this only pays off as a compressor on datasets “in the order of TBs (or more)”.
That shows a trained model works as a compressor. It doesn’t show the weights are a zip file of the training set, and they aren’t one. Training rewards whatever helps predict the next token: grammar, facts that show up again and again, how arguments and code are built. What’s repeated gets kept. What’s one-off mostly doesn’t.
Mostly. Models do reproduce some training text word for word. One study pulled hundreds of verbatim sequences out of GPT-2, including some that appeared in just one training document. Even then, the text comes out of the same multiplications.
How much can the weights hold? There’s no compression ratio to compute for Qwen3.8-Max: the config doesn’t say how much text it was trained on. One 2025 study measured GPT-style models at about 3.6 bits of capacity per parameter. Its largest model had 1.5 billion parameters, so stretching it this far is illustrative only: 2.45 trillion × 3.6 bits ≈ 1.10 TB.
Only 0.08% of the model is a lookup table
Part 1 called the embedding matrix a lookup table, and it is: token 7 selects row 7. That’s
parameters, or 0.08% of the model.
It’s the only part that works that way. Even its mirror, the output head, has the same shape but multiplies: it scores the final vector against all 248,320 rows. The other 99.92% of the model only transforms.
And that 0.08% doesn’t hold anything you could read. The row for “blue” is 8,192 numbers. None of them means “a color”.
248,320 tokens, one 8,192-wide vector
Here’s the number that breaks the database picture.
The residual stream is 8,192 numbers wide. A space like that has at most 8,192 directions that are fully independent of each other. At the last layer, that one vector has to tell 248,320 tokens apart:
That’s 30 times more tokens than independent directions. And tokens are the easy part. The model also has to represent ideas about them.
So nothing gets its own slot. Everything shares.
Researchers call this superposition. Give each feature a direction that’s almost independent of the others, and you fit far more features than dimensions. It works because most features are absent most of the time. The price is a little interference between them.
A database keeps records apart. A model overlaps them.
An expert is three matrices
Most of Qwen sits in its experts, so that’s where most of what it knows has to be.
The config gives each expert two sizes: 8,192 wide on the outside, 2,048 on the inside. Part 1 drew that as 8,192 → 2,048 → 8,192. In a gated block that’s three matrices, two going in (one of them is the gate) and one coming out:
There are 512 × 92 = 47,104 routed experts, and that’s the 2.37 trillion parameters in the first row of Part 1’s table.
Here’s the nearest thing to storage in the whole model. Each of an expert’s 2,048 hidden units behaves like a slot: a pattern it responds to (its key) and a vector it adds to the residual stream when it fires (its value). That’s 3 × 8,192 = 24,576 numbers per slot.
| Slots | |
|---|---|
| One expert | 2,048 |
| One layer (513 experts) | 1,050,624 |
| All 92 layers | 96,657,408 |
| Used by one token | 2,072,576 |
About a million slots per layer, counting the 512 routed experts and the shared one. One token runs 11 experts in each layer (10 routed plus the shared one), so it uses about 2% of them.
It sounds like a key-value store. But keys match by similarity, so every slot responds a little or a lot, and nothing ever returns “not found”. And the output is a sum. Thousands of slots each nudge the vector.
How close does it get to a record? Close enough to edit. Researchers changed which city a model places the Eiffel Tower in. They treated one layer’s matrix as a key-value store and wrote in a new pair. That’s as close to a record as it gets, and it was still a rank-one change to a matrix. No row was overwritten.
You can’t load half a model
Now the part that matters once you have to move the bytes.
There’s no index. Nobody can fetch the physics rows for a physics question. Past the embedding, the only selection is the router picking 10 of 512 experts, per token, per layer. And Part 1 showed that with 256 tokens in a step, 99.4% of experts get used anyway (assuming the router spreads tokens evenly).
The numbers are approximate, so they can be rounded. Cut four bits off every field in a database and you’ve corrupted it. Round a weight to fewer bits and you’ve added a little noise. Think of it as more of the interference the model already lives with. That’s why Kimi K3 can ship its experts in 4-bit: more parameters than Qwen, and about a third fewer bytes of weights.
After that, every byte has to arrive exactly. Rounding is a choice made once, on purpose. Storage doesn’t get to make it again. It can’t tell an important weight from an unimportant one, so all 2.50 TB has to land intact before the GPU produces a token. And ordinary compression barely helps: in one test, gzip got a small transformer’s 16-bit weights down to just 92.2% of their size. The bytes aren’t random, though. ZipNN, a lossless compressor built for model files, reports often saving 33%. Whether an FP8 model like this one shrinks that much is a question for Part 4.
Where this goes next
This is Part 2 of Signal & Weight. Part 3 picks up the rounding: quantization, the dial that decides how many bytes each weight takes and, with it, how many GPUs you need.
To see how precision changes the hardware across real models, open Inside Open Models and switch between Qwen3.8-Max (FP8) and Kimi K3 (4-bit experts).
Sources: the published config.json for Qwen3.8-2.4T-A95B, for the sizes behind every parameter, expert and slot count here (all computed from the config), and the tensor sizes in its FP8 release for the byte counts. Compression figures, including gzip on model weights: Delétang et al., Language Modeling Is Compression (2023), Table 1 of the current version; its abstract quotes slightly different image and audio rates. Verbatim memorization: Carlini et al., Extracting Training Data from Large Language Models (2020). Capacity per parameter: Morris et al., How much do language models memorize? (2025). Superposition: Elhage et al., Toy Models of Superposition (2022). Feed-forward layers as key-value memories: Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories (2020). Editing a fact in one matrix: Meng et al., Locating and Editing Factual Associations in GPT (2022). Lossless compression of model files: Hershcovitch et al., ZipNN (2024). The “slot” reading of an expert assumes a gated feed-forward block with three bias-free matrices. The 99.4% figure assumes the router spreads tokens evenly across experts. The 1.10 TB capacity figure is illustrative.
Next in Signal & Weight: Quantization: the dial that decides how many GPUs you need