Weights at rest: files, shards and formats

Qwen’s FP8 release of Qwen3.8-Max is 224 files on Hugging Face. 213 of them are weights, 2,496,102,697,232 bytes in all. The other 11 (config, tokenizer, chat template, licence and an index) add 51.7 MB.

Part 3 promised two answers about those bytes: how they’re split into files, and whether an FP8 or MXFP4 file still compresses, the question Part 2 left open. To answer them I read the header of every weight file in three releases, Qwen in FP8, Qwen in BF16 and Kimi K3, 522 files and about 9 TB in all. Then I pulled 300 random 1 MiB blocks out of each release and compressed them. None of this needed a full download, for a reason the first section explains.

A JSON header, then raw bytes

Take model-00005-of-00213.safetensors, one of Qwen’s FP8 files. It’s 17,182,226,672 bytes. The first 8 bytes are a number, 260,328, the length of the JSON text that follows. That text lists the file’s 2,048 tensors, each with a type, a shape and a byte range:

"model.layers.10.mlp.experts.0.gate_proj.weight": {
  "dtype": "F8_E4M3",
  "shape": [2048, 8192],
  "data_offsets": [2097152, 18874368]
}

After the header come the tensors’ raw bytes, back to back. That’s the whole format. In all 522 files the size works out exactly: 8 bytes, plus the header, plus the last tensor’s end offset. There are no gaps between tensors. The only padding is a few trailing spaces that bring each header to a multiple of 8 bytes.

Safetensors was written to replace PyTorch’s pickle files, which can run arbitrary code when loaded. Its header is plain JSON, capped at 100 MB. The byte ranges matter as much as the safety. A program that wants expert 0’s gate projection reads 16.8 MB at a known position and skips the rest of the file. That’s how I measured all of this: an HTTP range request, which asks the server for only a given span of bytes, for each header, then range requests for the sample blocks.

The headers are a small part of the total. All 213 of Qwen’s FP8 headers come to 36,444,688 bytes, 0.0015% of the release. Kimi K3’s 96 headers come to 75,766,584 bytes, the 76 MB that Part 3 had left over after counting Kimi’s tensors.

How 2.5 TB is cut into 213 files

Qwen’s files follow the model’s structure. Each of the 92 layers, plus the multi-token prediction (MTP) module, an extra layer that drafts tokens ahead for speculative decoding, gets two files of experts. One holds the gate and up projections of all 512 experts, 16 GiB of FP8 weights plus 2 MiB of scales. The other holds their down projections, 8 GiB plus 1 MiB. That’s 186 files. The other 27, from 0.5 to 3.8 GiB, hold everything every token uses: attention, DeltaNet, the shared experts, norms and the 4 GB embedding table.

The BF16 release has the same 213 files with the same contents. Its expert files hold exactly twice the weight bytes, 32 and 16 GiB, and no scales. Inside, it’s laid out very differently. BF16 stores each layer’s 512 experts as one fused tensor of shape [512, 4096, 8192]. At 34,359,738,368 bytes it’s the largest tensor in any of these files, and it gets a file to itself. The whole BF16 release has 1,609 tensors. The FP8 release has 287,119, because every expert matrix becomes its own tensor plus a tensor of scales, one BF16 number for each 128 × 128 block. The index that maps tensor names to files grows from 138 KB to 28.7 MB.

Kimi K3 cuts differently again: 96 files, 93 of which hold exactly one layer each. The first is the dense layer 0, at 2.2 GiB. The other 92 are MoE layers of 15.4 to 15.8 GiB, where a layer’s 4-bit expert weights, their scales and its BF16 attention all sit in one file. Its 497,220 tensors are the most of the three.

The layout matters when the model is loaded. Qwen’s layer 10 is spread over files 5, 6 and 11. Split the experts across 16 GPUs, 32 per GPU (expert parallelism), and the first GPU’s experts live in 186 of the 213 files, since every expert file holds all 512. Add the 27 files of shared weights that each GPU typically loads, and it touches all 213. For Kimi, whose layers have 896 experts each, 56 per GPU, the experts alone are in 92 of 96 files. However the files are cut, every GPU ends up reading a slice of nearly every file. That’s the subject of Part 5.

What the file doesn’t know

Safetensors has a type for Qwen’s FP8 weights, F8_E4M3: 1 sign bit, 4 exponent bits and 3 mantissa bits, with 448 as the largest value. It has none for Kimi’s 4-bit ones. Kimi’s expert tensors are typed U8, plain bytes:

"language_model.model.layers.20.block_sparse_moe.experts.411.w1.weight_packed": {
  "dtype": "U8",
  "shape": [3072, 1792]
}

3,072 × 1,792 bytes hold 3,072 × 3,584 weights, two to a byte. The first byte of that tensor is 0x29. Its high half, 0010, is the FP4 code for +1, and its low half, 1001, is the code for −0.5. Next to it is a second tensor, weight_scale, also U8, with one byte for each group of 32 weights. That byte is 121, which in E8M0 (8 exponent bits and no mantissa, so every scale is a power of two) means 2^(121−127) = 2⁻⁶. So the two stored weights are 0.015625 and −0.0078125. Which comes first is set by the packing code. Compressed-tensors, the library named in Kimi’s config, puts the first weight in the low half, so the tensor starts with −0.0078125, then 0.015625.

None of that is in the file. The header says “bytes”. It takes config.json, which declares mxfp4-pack-quantized with a group size of 32, and code that knows the packing order, to turn them back into weights.

The scales take real space: one byte per 32 weights comes to 85 GB of Kimi’s 1.56 TB. They hold very little information. Across 2,064,384 scale bytes from the w1 scale tensors of six sampled experts, 86.2% are 121 and 13.1% are 120.

Does it compress?

I took 300 random 1 MiB blocks from each release, spread across the files in proportion to their bytes, and compressed each with gzip (level 6) and zstd (level 3). For BF16 I also tried byte grouping: compressing the high and low byte of every 16-bit weight as two separate streams, the main trick of ZipNN, a compressor built for model files. Weighted by how many bytes each type of tensor holds, the results are:

Release	Size	zstd	Byte-grouped
Qwen3.8-Max, BF16	4.89 TB	79.1% · 3.87 TB	71.7% · 3.51 TB
Qwen3.8-Max, FP8 experts	2.50 TB	69.2% · 1.73 TB
Kimi K3, MXFP4 experts	1.56 TB	88.8% · 1.39 TB

The percentages are compressed size as a share of the original, so lower is better. Gzip lands within a point of zstd every time: 80.0%, 69.8% and 89.2%. They’re estimates from samples, with 95% intervals of about ±2 points for Qwen FP8 and ±0.3 points or less for the other two.

BF16 behaves the way the papers say it should. Of a 16-bit weight, 1 bit is the sign, 7 are the mantissa and 8 are the exponent. Only the exponent repeats much: in a sample of one attention matrix it carried 2.7 bits of information out of 8, close to the 2.6 bits reported for DFloat11, a lossless format that Huffman-codes the BF16 exponent. Compressed on its own with zstd, the byte that holds the sign and most of the exponent shrank to about 43% of its size. The other byte, almost all mantissa, barely shrank. ZipNN reports 66.4% for Llama 3.1 in BF16, where plain zstd manages 77.7%. Plain zstd does about the same on Qwen, 79.1%. My byte-grouped 71.7% is further behind ZipNN, which separates the exponent bits exactly and then uses Huffman coding: even its zstd variant reaches 68.8%.

Kimi’s 4-bit codes barely move. On their own they compress to 94.3%. The 16 codes carry 3.75 bits of information out of 4, so there is almost nothing left to remove. The release as a whole does better, 88.8%, because the scale bytes compress to about 15% and the BF16 attention weights to 78%. That’s in the range ZipNN found for models quantized with GPTQ and AWQ, 85% to 91%.

That leaves Qwen’s FP8 file. An 8-bit weight has only a 4-bit exponent and 3 mantissa bits, so it has less of the redundancy that makes BF16 compress. By that reasoning it should compress worse than BF16. It compresses better: 69% against 79%.

That also answers the question Part 2 left open. ZipNN reports often saving 33%. Qwen’s FP8 file saves 31% with plain zstd, close to that figure, but a large part of the saving comes from something specific to this model.

The zeros

That something is zeros. Of Qwen’s 300 sampled blocks, 289 fell in expert tensors, and on average 24% of their bytes encode zero, about half as +0 and half as −0. They’re unevenly spread: the median block is 8.7% zeros, and a quarter of the blocks are more than 43% zeros. The fourth chapter of the interactive plots all 289 blocks, and they fall close to a straight line. Blocks with under 1% zeros compress to 82%. Blocks with more than half zeros compress to 39%. A straight line through the points crosses the no-zeros edge at about 83%. So without the zeros, the FP8 file would compress slightly worse than BF16, as the bit layout suggests, and the zeros account for about 14 of its 31 points of saving.

The BF16 release shows where they come from. There, 27.6% of the expert weights in my samples are smaller than 7.5 × 10⁻⁹, against a median of 0.004 for the rest. They’re tiny but not zero. In one expert I sampled from both releases, layer 61’s expert 490, the median BF16 weight is 1.6 × 10⁻¹⁶, and 91% of its FP8 bytes are zero. About one row in ten of the BF16 expert matrices I sampled is tiny from end to end.

FP8 rounds most of these to zero because of how its scales work. Each 128 × 128 block gets one scale, set so that its largest weight maps to 448, the top of the E4M3 range. I checked this against the BF16 release. In all 4,096 blocks of four experts sampled from both files, scale × 448 matches the block’s largest BF16 weight to within 0.33%, consistent with the FP8 file having been made from the same weights. The smallest non-zero value that can be stored is then about 1/229,000 of that largest weight. A weight of 10⁻¹⁶ in a block whose largest weight is 0.03 is far below that, so it’s stored as zero. A block where every weight is tiny gets a tiny scale and keeps its values. So the two counts don’t match expert by expert. In layer 86’s expert 92, 99.95% of the BF16 weights are tiny but only 12.4% of the FP8 bytes are zero. In layer 0’s expert 241, 40.3% are tiny and 45.4% are zero, because weights just above the cutoff are flushed too.

Why so much of Qwen’s expert capacity is nearly empty is something the files don’t say. Experts that the router seldom picks would look like this, but that’s a guess. What the files do show is that a large part of the 31% saving on Qwen’s FP8 file comes from this model’s weights. An FP8 model without the tiny weights would likely compress to around 82–83%, if its weights look like Qwen’s blocks without zeros.

What this means for storage

Rounding is the bigger lever. Going from BF16 to FP8 cut Qwen’s release by 49%. The best lossless result I got on the BF16 release, 3.51 TB, is still about a terabyte bigger than the FP8 release before any compression. After rounding to 4 bits, a compressor finds almost nothing.

What’s left is still worth something. Zstd at its fast default takes 0.77 TB off every copy of Qwen’s FP8 release. That’s a saving on every disk and network link that holds or moves it. But a GPU can’t compute on zstd blocks, so the bytes have to be decompressed somewhere between storage and GPU memory. DFloat11 does this on the GPU for BF16 models. For the FP8 file, much of the compressible part is the runs of zeros, and an engine that skips zero blocks or stores them sparsely could get the same saving without a general-purpose compressor. I haven’t tested either for this model.

Where this goes next

This is Part 4 of Signal & Weight. Part 5 is the cold start: how long it takes to get these files off storage and into GPU memory, what limits it, and what the layout described here does to it.

Step through the files, the bytes and the zeros in the interactive, or go back to the precision dial to see the rounding that produced them.

Sources: file listings and the safetensors header of every shard of Qwen3.8-2.4T-A95B-FP8, Qwen3.8-2.4T-A95B and Kimi-K3 on Hugging Face, read on 2026-10-16, plus each release’s config.json (Qwen’s weight_block_size of [128, 128]; Kimi’s compressed-tensors MXFP4 config). Format: safetensors README, for the layout, the 100 MB header limit and the comparison with pickle. FP4 and E8M0 encodings: the OCP Microscaling Formats (MX) v1.0 spec. Compression: Hershcovitch et al., ZipNN (2024), Tables II and III and Section VI-A; the compressed-tensors library, for the FP4 packing order; Zhang et al., 70% Size, 100% Accuracy (DFloat11) (2025). The compression figures are my measurements: 300 random 1 MiB blocks per release, read by HTTP range request, compressed with Apple gzip 479 at level 6 and zstd 1.5.7 at level 3, and weighted by each tensor type’s share of the release’s bytes. Kimi’s scale tensors are smaller than the sample blocks, so their 15% comes from six whole w1 scale tensors, and Kimi’s gzip figure reuses it. The 83% no-zero figure is a straight-line fit through the 289 FP8 blocks (r = −0.98). “Tiny” means a magnitude below 2⁻²⁷, about 7.5 × 10⁻⁹, read from the BF16 exponent. The suggestion that tiny weights belong to rarely-picked experts is a guess, not a measurement.