A model doesn't just sit in memory once and stay put. Here's the data flow from a stored checkpoint to a generated token, and why the memory footprint keeps growing well after the model has already loaded.
Most explanations of AI infrastructure treat "the model" as a fixed-size thing that either fits in memory or doesn't. In practice, inference has two very different memory problems stacked on top of each other: a large but fixed allocation for the model's weights, and a smaller-at-first but unbounded and growing allocation for everything the model remembers about the conversation it's currently having. The second one is what actually breaks capacity planning.
Five stages. Two of them are static. One of them is the reason this is a storage problem at all.
The trained checkpoint sits in object storage (S3, a blob store, a model registry), usually saved in FP16/BF16 or a quantized format (INT8/INT4) to cut its footprint. The rule of thumb is blunt but reliable: about 2GB per billion parameters at FP16, half that at INT8.
Llama 2 70B at FP16: 140GB of weights, before anything else has loadedAt server startup, weights are read once from storage into the GPU's high-bandwidth memory. This allocation doesn't move for as long as the model stays deployed: it's the "rent" a served model pays on HBM before a single request arrives.
An 8×H100 node has 640GB of HBM total. That 70B model's 140GB is already spoken for.As the model generates each token, it stores that token's key and value vectors in HBM so it never has to recompute them. Model weights are fixed size. This one is not: it scales with context length, batch size, and how many people you're serving at once. Grouped-query attention (sharing key/value heads across groups of query heads) is the main reason this number is survivable at all rather than several times larger.
Llama 3 70B, 128K context, one user: 2 × 80 layers × 8 KV heads × 128 head-dim × 131,072 tokens × 2 bytes ≈ 42.9GB, over half of one H100's HBM, before a second user connectsOnce the KV cache stops fitting in HBM, there are only two options: evict and recompute later (costs latency), or push it somewhere else and pull it back on demand, CPU RAM first, then NVMe over RDMA via GPUDirect Storage. This only pays off when fetching the cache back is faster than just recomputing it, which is exactly the trade-off a 4-tier memory hierarchy is built to manage.
Nvidia's ICMSP (announced CES 2026): up to 5x more tokens/sec and 5x better power efficiency, specifically for long-context workloads out to 1M tokensThe generated token goes back to the client. It's a small write compared to everything above it, but production systems still log requests and responses back to object storage for observability, auditing, and eval: a steady trickle, not a spike.
Stage 02 is the number most capacity planning still centers on: how many GPUs does it take to hold the weights. That number is knowable in advance and doesn't change once you've picked a model. Stage 03 is the one that actually determines whether a deployment falls over: it's a function of how people use the model, not the model itself, and it's the reason KV cache offload (Stage 04) turned from a niche optimization into a standard piece of inference infrastructure inside about a year.
This is also why "storage" and "AI infrastructure" have become the same beat. NVMe throughput, RDMA fabric speed, and CXL-attached pooled memory (see Issue 03's Uplevel Highlight) all show up in Stage 04 as directly as any GPU spec does in Stage 02.
One email a week on AI storage and memory infrastructure. No fluff, no spam.
One email a week. Unsubscribe link in every issue.