Quantization: the dial that decides how many GPUs you need

Kimi K3 has about 2.78 trillion parameters and its weights fill 1.56 TB. Qwen3.8-Max has 2.45 trillion and fills 2.50 TB. The bigger model is the smaller file because Moonshot stores most of Kimi’s weights in 4 bits, where Qwen uses 8.

Part 2 ended on the reason that’s possible at all. A weight is an approximate number, so it can be rounded. This part is about how far you can round it, what that costs, and why the saving on disk doesn’t carry through evenly to the GPUs.

One weight, rounded three times

Pick a weight from one of Qwen’s experts, say 0.0137. (The value is illustrative. The formats below are real.)

BF16, the 16-bit format models are usually trained in, stores it as 0.013672. It keeps 7 bits after the binary point, so the error is about 0.2%.

FP8, the format Qwen ships its experts in, keeps only 3. It happens to land on the same 0.013672, because that number is 1.75 × 2⁻⁷ and 1.75 needs only two bits after the point. Most weights aren’t that lucky, but FP8 is never off by more than about 6%.

FP4 is where the trouble starts. Its four bits give 16 patterns, and since +0 and −0 are the same number, only 15 distinct values: zero and ±0.5, 1, 1.5, 2, 3, 4 and 6. To represent anything else, each block of weights shares a scale. In the MXFP4 format, 32 weights share one power of two. If that scale is 2⁻⁷, our weight has to choose between 1.5 × 2⁻⁷ and 2 × 2⁻⁷. It becomes 0.015625, 14% off.

Qwen has 2,396,591,751,168 weights in its experts, 98% of the model. At each precision, the whole model comes to:

Experts stored as	Size	B200s for the weights
BF16 (and everything else BF16)	4.89 TB	32
FP8, as released	2.50 TB	16
MXFP4 (everything else BF16)	≈1.37 TB	9

The last row is hypothetical. Qwen doesn’t publish a 4-bit version, so I built it from the tensor counts of the FP8 release: the experts at 4.25 bits each, the other 49,590,974,336 weights left in BF16.

Why 4-bit costs 4.25 bits

The shared scale isn’t free. MXFP4, from the Open Compute Project’s microscaling spec, stores one 8-bit scale per 32 weights. That’s an extra quarter bit per weight, so 4.25 in total. NVIDIA’s NVFP4 uses smaller blocks of 16, each with an 8-bit FP8 scale, plus one 32-bit scale per tensor: 4.5 bits.

Kimi K3’s files let you check this. Its config.json declares MXFP4 with a group size of 32, and Hugging Face reports what’s in its tensors: 2,722,740,830,208 expert weights, 57,179,884,544 in BF16 and 11,122,432 in FP32. The files add up to 1,560,936,091,448 bytes. Take out the BF16 and FP32 tensors and divide what’s left by the expert count, and you get 0.53127 bytes per weight. That’s 4.250 bits, with about 76 MB to spare for file headers.

OpenAI describes gpt-oss the same way: its expert weights are “quantized to 4.25 bits per parameter”.

What a shared scale does with an outlier

A shared scale has to fit the largest number in its block. Most of the time that’s fine. The problem is the rare weight, or more often the rare activation, that is far larger than everything around it.

Large language models are known to grow these. The LLM.int8() paper found features up to 20 times larger than the rest. At around 6.7 billion parameters they appeared in every layer, concentrated in just 6 of the hidden dimensions. With a single scale per row, one of those values forces the scale up, and the ordinary values next to it round down to zero.

The third chapter of the interactive runs that experiment on 256 illustrative weights with one outlier. With one scale for all 256, the error on the ordinary weights is 84%, and 226 of the 255 are stored as zero. With one scale per 32, as in MXFP4, the damage stays inside the outlier’s block and the error drops to 31%.

The older fix is to rotate first. Multiply the weights by an orthogonal matrix, quantize, then undo the rotation. The outlier gets spread thinly across every dimension, so no single block has to stretch around it. QuaRot used this to run Llama 2 70B with weights, activations and cache all in 4 bits, losing at most 0.47 points of WikiText-2 perplexity and keeping 99% of its zero-shot accuracy.

In the interactive, rotating before MXFP4 only takes the error from 31% to 26%. A 2025 study of FP4 formats found that “NVFP4’s small group size provably neutralizes traditional outlier mitigation techniques”. Small blocks already contain most of the damage, so there’s less left for rotation to fix. The same study found that MXFP4’s power-of-two scales cause errors of their own, and proposed a block-wise rotation that brings MXFP4 close to NVFP4. Its conclusion: “FP4 is not an automatic upgrade over INT4.”

Rotation is also how Google’s TurboQuant compresses the KV cache, the memory a model keeps for every token of a conversation. The paper reports “absolute quality neutrality with 3.5 bits per channel” and more than 5× compression, measured on Llama 3.1 8B and Ministral 7B. Google’s blog post about it goes further, claiming at least 6× and an 8× speedup on H100s. The paper doesn’t report those figures, and its experiments ran on an A100. The cache is Part 6.

Which weights get rounded

Kimi K3’s config also lists what it leaves alone: attention, the shared experts, the dense first layer, the output head. Only the routed experts go to 4 bits. Qwen’s FP8 release draws the same line, one bit-width up. The experts hold nearly all of the parameters, and the parts every token passes through stay at 16 bits.

That choice decides how much of the saving reaches the GPU. On disk, 4-bit experts take Qwen from 2.50 to about 1.37 TB, 45% less. Per token, the picture is different. One token reads 10 routed experts in each of 92 layers, about 46 GB in FP8, plus about 98 GB of BF16 weights that every token uses (counting the whole embedding table, as Part 1 did). Rounding the experts to 4 bits cuts the first number, not the second. One token’s read drops from about 144 GB to 123 GB, only 15%.

A busy server is closer to the 45%. Part 1 worked out that with 256 tokens in a step, 99.4% of the experts get used, so the server reads nearly the whole model each step anyway.

Servers come in eights

Halving the bytes doesn’t halve the hardware, because the hardware comes in boxes. An 8-GPU DGX B200 has 1,440 GB of GPU memory. Qwen in FP8 needs two of those. At ≈1.37 TB, the 4-bit version would need nine GPUs’ worth of usable memory, which is still two boxes.

The next generation changes the answer. NVIDIA lists 2.1 TB for an 8-GPU DGX B300, enough to hold the 4-bit Qwen in one box with room to spare.

Where the saved memory goes

If you keep the two B200 servers, the freed memory doesn’t sit empty. It goes to the cache, and the cache is what limits how many conversations a server can hold.

Part 1 put Qwen’s cache at 12.9 GB for one conversation at 128K tokens. In FP8, 16 B200s keep about 156 GB of weights each, which leaves 384 GB across the set: room for about 29 of those conversations. With 4-bit experts, the weights drop to about 86 GB per GPU, which leaves about 1.5 TB, room for about 116.

Those numbers ignore the memory a serving engine needs for its own working space, so they’re upper bounds. The ratio is the point. On the same GPUs, rounding the experts from 8 bits to 4 makes room for roughly four times as many long conversations.

What gets lost

This is the part with the least certainty, because quality loss depends on the model, the method and what you measure.

Rounding a finished model, with nothing retrained, holds up well at 4 bits. On Llama 2 70B, AWQ’s 4-bit weights move WikiText-2 perplexity from 3.32 to 3.41. A large evaluation of quantized Llama 3.1 models found FP8 “effectively lossless” and 4-bit weights recovering 99.36% of the original accuracy on average. NVIDIA reports that converting DeepSeek-R1 from FP8 to NVFP4 cost “1% or less” on its benchmarks.

The bigger change is that labs now train for the format. Moonshot says Kimi K3 “applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations”. For Kimi K2 Thinking, it reported every benchmark result at INT4. gpt-oss was post-trained with its experts in MXFP4. When a model is trained at the precision it ships in, the published benchmarks are already measured at that precision, and nobody publishes the full-precision comparison. So for these models, the honest answer to “what did 4 bits cost?” is that the labs don’t say.

Where this goes next

This is Part 3 of Signal & Weight. Part 4 follows the bytes onto storage: how 1.56 or 2.50 TB gets split into files and shards, and whether an FP8 or MXFP4 file still compresses, the question Part 2 left open.

To see the dial for yourself, step through the precision dial, or open Inside Open Models and compare Qwen3.8-Max (FP8) with Kimi K3 (MXFP4 experts).

Sources: tensor counts and file sizes for Qwen3.8-2.4T-A95B-FP8 and Kimi-K3 from Hugging Face, plus Kimi K3’s config.json and model card (Section 4). Formats: the OCP Microscaling Formats (MX) v1.0 spec and NVIDIA, Introducing NVFP4 (2025), also the source of the DeepSeek-R1 figure. gpt-oss: OpenAI, gpt-oss-120b & gpt-oss-20b model card (2025). Outliers: Dettmers et al., LLM.int8() (2022). Rotation: Ashkboos et al., QuaRot (2024). FP4 and rotation: Egiazarian et al., Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization (2025). KV cache: Zandieh et al., TurboQuant (2025), and Google Research’s blog post (2026). Quality: Lin et al., AWQ (2024), Table 4; “Give Me BF16 or Give Me Death” (2025); Moonshot AI, Kimi K2 Thinking model card. Hardware: NVIDIA’s DGX B200 and DGX B300 pages. B200 counts assume about 153 GB per GPU for weights, as in Part 1, except FP8, which follows Qwen’s 16-GPU recipe. The 4-bit Qwen sizes, per-token bytes and conversation counts are my arithmetic and approximate. The outlier experiment uses illustrative weights.