Serving a model needs one copy of its weights. Training the same model needs several, plus a running record of everything the forward pass computed, all before a single gradient gets applied.
The inference guide covers what happens once a model is trained and deployed. This one covers the much heavier phase before that: pretraining, where the memory and storage footprint of a model is not 2 bytes per parameter but closer to 16, and where a single failed GPU can threaten weeks of compute if the storage layer underneath it is not built for fast recovery.
Five stages. One of them alone is bigger than a whole training node's memory.
Pretraining starts with a petabyte-scale raw crawl, text, code, scientific journals, and more, that gets filtered hard before it ever reaches a GPU. What survives is measured in tokens, not bytes. The filtered corpus still lives in object storage throughout training, sharded and streamed to nodes rather than copied wholesale.
Llama 3.1 pretraining corpus: roughly 15 trillion tokens, filtered down from a petabyte-scale raw crawlTraining keeps three copies of state per parameter where inference keeps one: the parameters, their gradients, and the optimizer's own bookkeeping. With Adam in mixed precision, that is BF16 copies of parameters and gradients (2 bytes each) plus FP32 master weights, momentum, and variance for the optimizer (4 bytes each): 16 bytes per parameter, against 2 bytes per parameter for inference weights alone.
Same 70B model: 1.12TB of model states at 16 bytes/parameter. That is 8x the 140GB inference footprint, and more than an entire 8xH100 node's 640GB of HBM, before a single activation is computedEvery layer's forward pass produces activations that have to be kept around for the backward pass to compute gradients from, unless they get recomputed on the fly instead. This scales with batch size, sequence length, and depth, not parameter count, which is why the same model that trains fine at one batch size runs out of memory at another. It is training's version of inference's KV cache: the number nobody can precompute once and forget.
Roughly batch × layers × (5 × seq_len × hidden_dim + 2 × heads × seq_len² + 2 × seq_len × FFN_dim) bytes, per common estimators. That seq_len² term means doubling sequence length does worse than double this numberTraining runs periodically flush model states to storage so a hardware failure does not erase days of progress, using the same tiered pattern covered in Issue 01's optimization playbook: fast writes to node-local NVMe every few minutes, propagation to shared storage roughly every 30 minutes, and a durable copy in object storage every few hours. What changes at scale is how much each GPU actually writes: spread across enough data-parallel GPUs, each one only owns a shard of the model state, so per-GPU checkpoint size drops sharply as the job scales out.
13B model checkpoint, per GPU: ~10.4GB at data-parallel degree 1, ~650MB at degree 16. A 30B model: ~13.8GB down to ~870MB over the same rangeWhen a node fails, and at large enough scale something is always failing, the replacement reads the most recent checkpoint back from storage and resumes instead of restarting the run. The faster storage can serve that read, the less compute time an outage actually costs, which is why checkpoint I/O bandwidth gets engineered as carefully as the training compute itself.
Stage 02's number is the one that actually explains why frontier model training spans thousands of GPUs rather than a handful. A single 8xH100 node cannot hold a 70B model's training state at all, let alone the activation memory on top of it. That gap is what tensor parallelism, pipeline parallelism, and ZeRO-style optimizer sharding exist to close, splitting the 1.12TB of model states across enough GPUs that each one only has to hold a slice.
The storage layer feels this indirectly but constantly: every one of those parallelism strategies still has to checkpoint its shard of the state on the same schedule, and the whole job stalls if any one of those writes falls behind. That is the throughline from Stage 04 back to the tiered checkpointing already covered in Issue 01, now at a scale where the storage system, not the GPUs, decides how much of an outage actually costs.
One email a week on AI storage and memory infrastructure. No fluff, no spam.
One email a week. Unsubscribe link in every issue.