NVMe KV Cache Offloading for LLM Inference: Serve 10x More Users on One H100

TL;DR

NVMe KV cache offloading lets you serve 10x more concurrent users on a single H100 by moving cold KV cache blocks from GPU HBM to NVMe SSD, effectively turning fast local storage into a third tier alongside GPU memory and CPU RAM. Instead of hitting the GPU memory wall at 128K+ context lengths, vLLM keeps active generation blocks in HBM, hot conversation history in CPU DRAM, and cold prefix cache on NVMe — using LMCache as the software bridge. The result is 2x throughput and TTFT dropping from ~11 seconds to ~1.5 seconds for re-encountered system prompts, with cost per user falling from $2.63/hr to roughly $0.29/hr.

1. The GPU Memory Wall: Why KV Cache Is the New Bottleneck

The KV cache is the silent capacity killer in LLM inference. Every token a model reads or generates has a corresponding key-value pair stored in GPU HBM so the forward pass skips recomputation. Those pairs scale with the number of layers, attention heads, head dimension, sequence length, and batch size — all simultaneously. The math is unforgiving: at a 128K context length, a single Llama 3.1 70B user consumes roughly 40 GB of KV cache in BF16. An H100 with 80 GB of HBM can therefore accommodate only one or two such users at a time. Push to 1M tokens and the per-user cache balloons to approximately 320 GB — exceeding even the B200’s 192 GB of onboard HBM [1][2].

In-GPU mitigations buy breathing room but do not remove the wall. PagedAttention, introduced by the vLLM team, shards the KV cache into fixed-size blocks and eliminates internal fragmentation, effectively reclaiming the overhead that would otherwise force overprovisioning [1]. FP8 quantization halves KV cache footprint compared to BF16 at negligible quality loss for most generation workloads [2]. vLLM’s --swap-space flag allows blocks to spill from HBM into CPU DRAM, providing a safety valve for bursty traffic [1]. Each technique helps in isolation; together they still leave long-context, multi-user serving as a fundamentally undersupplied problem.

The conclusion is straightforward: if your workload combines context lengths above 32K with multiple concurrent users — or any scenario where the prefix (system prompt, RAG preamble, conversation history) is shared and re-encountered — GPU-local memory will always be the binding constraint. NVMe offloading is the only practical path to multi-user serving at scale, because it decouples cache capacity from the GPU’s onboard memory entirely [1][2].

2. The Three-Tier Storage Hierarchy: GPU HBM → CPU DRAM → NVMe

Effective KV cache offloading treats storage as a hierarchy, not a binary on/off switch. The guiding principle is simple: keep what you need now on the fastest tier, demote what you need later to slower tiers, and promote it back when the conversation circles around.

Hot tier — GPU HBM (~3.35 TB/s bandwidth, sub-microsecond latency). Active generation blocks — the keys and values for tokens being produced right now — live here. PagedAttention keeps them as compact as possible, but this tier is still the scarcest resource. A typical H100 80 GB can dedicate roughly 90% to KV cache (the remainder goes to weights and activations), yielding about 72 GB of usable cache space.

Warm tier — CPU DRAM (~63 GB/s over PCIe 5.0, ~10–100 µs latency). Multi-turn conversation history sits here. When a user’s prompt re-enters the context window, the system promotes the relevant KV blocks from NVMe up to CPU RAM first, avoiding a direct NVMe-to-GPU transfer. The PCIe 5.0 x16 link between CPU and GPU provides a high-bandwidth staging area — fast enough that the promotion penalty is mostly absorbed.

Cold tier — NVMe SSD (~7 GB/s over PCIe 4.0, ~100 µs to 1 ms latency). Historical context and prefix cache live here permanently until evicted by capacity pressure. Loading a 40 GB cache from NVMe takes roughly 6 seconds — slow compared to in-GPU access, but dramatically faster than recomputing the 128K-token prefill, which costs approximately 11 seconds on an H100 [1][2]. That speed differential is the entire economic argument for offloading.

The offloading heuristic is straightforward: blocks actively participating in generation stay in HBM; blocks for recent turns that are about to fall off the attention window move to CPU RAM; blocks for earlier turns or infrequently re-encountered prefixes migrate to NVMe. The system promotes them back up the chain as needed, creating a fluid data lifecycle that matches actual access patterns rather than a static allocation.

3. NVIDIA ICMSP / CMX: Hardware-Accelerated KV Cache Tiering

NVIDIA’s answer to the KV cache capacity problem is ICMSP — officially branded as CMX (Context Memory eXtension) — announced at CES 2026. ICMSP moves the tiering logic from software into dedicated hardware, and the architecture has three key components [1][2].

First, cuFile is a GPU-direct NVMe transfer API that lets the GPU read and write NVMe storage without routing data through CPU memory. This eliminates the interrupt overhead and CPU context-switch penalty that traditionally makes GPU-to-storage transfers expensive. The GPU’s DMA engine talks directly to the NVMe controller.

Second, the BlueField-4 DPU (Data Processing Unit) offloads I/O scheduling from the host CPU. Instead of the host’s SMs juggling memory management alongside compute work, the DPU manages the tiering decisions — deciding which blocks to evict from HBM, which to stage into CPU RAM, and which to write to NVMe. The host GPU is free to focus on inference.

Third, the NIXL protocol (NVIDIA eXchange Interface) provides a unified abstraction over four transport layers: NVLink for single-node transfers, InfiniBand RDMA for multi-node clustering, PCIe for local tiering, and a TCP fallback for environments without RDMA-capable networking. This means the same tiering semantics apply whether you’re running on a single GPU or across a distributed cluster [1][2].

Storage partners include VAST Data, Dell, HPE, Pure Storage, Supermicro, and WEKA — all validated with NVIDIA’s reference implementation [1][2]. The important caveat: hardware acceleration via BlueField-4 DPUs and cuFile is not available on standard bare-metal GPU nodes today. For teams without access to this hardware, LMCache provides the same three-tier pattern entirely in software.

4. LMCache: Software-Only NVMe Offloading for vLLM

LMCache is the open-source engine that makes the three-tier hierarchy practical without specialized hardware. It plugs into vLLM, SGLang, and NVIDIA Dynamo as a KV cache engine with configurable persistent storage backends [1][2].

The key distinction between LMCache and vLLM’s built-in --swap-space is persistence and sharing. vLLM’s swap space moves blocks from HBM into CPU RAM only — it disappears on instance restart, and it is scoped to a single vLLM process. LMCache persists cache blocks to NVMe local disk, Redis, or remote object stores, survives restarts, and is shared across multi-replica clusters [1][2]. This makes it suitable for production workloads where prefix cache hits span multiple service instances.

Configuration centers on a YAML file (lmcache_config.yaml) where you declare the tiers you want: local NVMe disk, CPU RAM, Redis, or S3-compatible object stores. Setting local_disk: true with a path and max size turns on the cold tier; setting local_cpu: false (the default for GPU-heavy workloads) skips the warm tier if your CPU memory is constrained [1][2].

Production validation comes from multiple sources. Google Cloud has benchmarked LMCache on GKE Inference, CoreWeave uses it in production, and Cohere has published internal benchmarks [1][2]. VAST Data’s published results are the most concrete: for a 128K-token system prompt on an H100, TTFT drops from approximately 11 seconds (cold) to 1.4–1.5 seconds on cache hit — a 7.5× improvement that compounds when you consider the throughput gains from serving more concurrent users [1][2].

5. Step-by-Step: Deploy LMCache with NVMe Offloading on vLLM

This section walks through getting LMCache running with NVMe offloading on a standard bare-metal GPU instance [1][2].

Step 1 — Provision the instance. Use a bare-metal GPU provider that includes NVMe storage at no extra cost. Spheron’s H100, H200, and B200 instances all ship with NVMe drives included. The NVMe device should be formatted and mounted before installation (verify with lsblk, nvme list, and df -h).

Step 2 — Install dependencies. Run pip install vllm lmcache and verify version compatibility at docs.lmcache.ai. The LMCache and vLLM version pairs matter — mismatched versions are the most common source of silent failures [1][2].

Step 3 — Configure the cache tiers. Create an lmcache_config.yaml:

local_disk: true
local_disk_path: /mnt/nvme/lmcache   # your mounted NVMe path
max_local_disk_size: 200              # GB — set based on your workload
local_cpu: false                       # skip warm tier for GPU-heavy workloads

Adjust max_local_disk_size to balance cache hit rates against NVMe wear and capacity. A 200 GB pool handles roughly 5 full 128K KV caches in BF16 [1][2].

Step 4 — Launch vLLM. Start the server with offloading-aware flags:

vllm serve your-model \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --swap-space 16 \
  --lmcache-config lmcache_config.yaml

The --kv-cache-dtype fp8 flag halves the KV cache footprint compared to BF16, effectively doubling your concurrent user capacity [1][2]. --enable-prefix-caching activates shared prefix caching so system prompts are reused across requests. --gpu-memory-utilization 0.90 reserves 90% of HBM for cache (10% for weights and activations). --swap-space 16 gives 16 GB of CPU RAM as an overflow safety valve [1][2].

Step 5 — Validate. Send a batch of requests sharing a long prefix, then repeat them. Check that the NVMe directory grows as the cache populates and that the second request’s TTFT drops from ~11 seconds to ~1.5 seconds — confirming the cache hit path is working [1][2].

For B200 instances, add --kv-cache-dtype nvfp4 to enable NVIDIA’s NVFP4 quantization, which makes KV cache 75% smaller than BF16 even beyond FP8 savings [1][2].

6. Benchmarks, Costs, and When to Offload (or Skip It)

The economic case for NVMe KV cache offloading is clearest when measured in cost per concurrent user [1][2].

Throughput. At 8 concurrent users with 128K context on an H100, LMCache combined with FP8 KV cache boosts throughput from approximately 900 tokens per second to roughly 1,800 tokens per second — a 2× improvement. The gain comes not from NVMe I/O speed, but from avoiding prefill recomputation. When a cache hit occurs, the system skips the expensive prefix re-encoding entirely and proceeds directly to decoding [1][2].

TTFT reduction. Warm prefix TTFT drops from ~11 seconds (cold, full prefill) to ~1.4–1.5 seconds (cache hit) for 128K-token system prompts. Even the ~6-second NVMe load time is faster than recomputing the prefix [1][2].

Cost per user. An H100 instance costs roughly $2.63/hour. Without offloading, that’s ~$2.63 per user when serving one user. With LMCache + FP8 KV and 8–10 concurrent users, the effective cost falls to approximately $0.29 per user per hour — a 9× reduction [1][2].

When to use NVMe offloading:

• Context lengths above 32K per user
• Multi-turn chatbots with recurring conversation history
• Shared-prefix RAG pipelines where every query starts with the same system prompt and document block
• Batch inference with common preambles (fine for report generation, code summarization)

When to skip it:

• Short-context workloads under 4K tokens (the cache overhead isn’t worth the complexity)
• Unique-prefix workloads with near-zero cache hit rates (no benefit from storage)
• Strict p99 latency SLAs under 100ms (NVMe load times violate the budget)
• Speculative decoding scenarios (the pattern of cache misses during speculation can degrade performance)

The rule of thumb: if your prefix cache hit rate is above 20% and your context exceeds 32K, NVMe offloading pays for itself. Below those thresholds, the operational overhead usually outweighs the benefit [1][2].

Conclusion

The KV cache is no longer a minor optimization consideration — it has become the primary capacity constraint on LLM inference at scale. As context windows expand to 128K, 1M, and beyond, the math of BF16 key-value storage turns GPU HBM into a hard ceiling that no amount of software trickery alone can meaningfully raise. PagedAttention, FP8 quantization, and CPU swap space each buy incremental headroom, but they share a common limitation: they operate within the GPU’s onboard memory boundary. NVMe offloading breaks that boundary, treating persistent storage not as an afterthought but as a first-class tier in the KV cache lifecycle.

The path forward splits into two complementary approaches. NVIDIA’s ICMSP/CMX moves tiering decisions into dedicated hardware — cuFile for GPU-direct NVMe transfers, BlueField-4 DPUs for I/O scheduling, and NIXL for unified transport abstraction — offering the lowest-latency path for organizations that can access the required infrastructure. LMCache, by contrast, delivers the same three-tier pattern entirely in software, plugging into vLLM and SGLang with configurable backends ranging from local NVMe to Redis to S3. Both converge on the same insight: the most effective KV cache strategy matches storage speed to access frequency, keeping active generation blocks in HBM, recent conversation turns in CPU RAM, and historical prefixes on NVMe.

The economics are compelling. A 7.5× TTFT improvement on prefix cache hits, combined with a 9× reduction in cost-per-user at scale, transforms long-context serving from a capacity-constrained exercise into a predictable, repeatable workload. The decision to adopt is not binary — it hinges on your specific mix of context length, prefix overlap, and latency SLA. Above 32K context with a prefix hit rate exceeding 20%, the ROI is clear. Below those thresholds, the complexity is hard to justify.

As the LLM inference stack matures, the KV cache will stop being an afterthought and start being a first-class resource — scheduled, tiered, and shared across replicas the way weights and activations already are. The question is no longer whether to offload, but how aggressively to optimize the path from GPU to NVMe and back. For teams serving long-context workloads at production scale, the answer is becoming obvious.

Methodology

• Data checked: 2026-07-08
• Sources consulted: Spheron Network blog posts, LMCache technical report
• Assumptions: Benchmarks reflect VAST Data and CoreWeave published results; actual performance varies by model size, batch configuration, and storage hardware.
• Limitations: This guide covers GPU-based LLM inference only and does not address multi-node distributed KV cache sharing beyond cluster-level prefix replication.
• Jurisdiction: Global.

Source list

• Spheron Network, “NVMe KV Cache Offloading for LLM Inference” — https://www.spheron.network/blog/nvme-kv-cache-offloading-llm-inference/ (accessed 2026-07-08)
• Spheron Network, “NVIDIA ICMSP KV Cache NVMe Inference Guide” — https://www.spheron.network/blog/nvidia-icmsp-kv-cache-nvme-inference-guide/ (accessed 2026-07-08)
• LMCache, “KV Cache Layer for Enterprise-Scale LLM Inference” (tech report) — https://lmcache.ai/tech_report.pdf (accessed 2026-07-08)

Trust Stack

• AI draft model: qwen3.6:35b
• AI review model: qwen3.6:35b
• Human editorial review: No (automated factory pipeline)
• Last substantive check: 2026-07-08
• Corrections policy: If you spot an error, contact us via the Contact page
• Affiliation: theLLMs has no vendor affiliation, sponsorship, or commercial relationship with any AI provider mentioned

Related guides

• LLM Serving Benchmarks
• vLLM Inference Engine Guide

Change log

• 2026-07-08: first published