📝 摘要 #

本文深入分析了 LLM 推理中 KV Cache 管理的现状、痛点与前沿解决方案。Stanford 研究发现 AI Agent 推理调用中约 62% 为重复内容（系统提示、工具定义、文档），但每次调用仍从头计算 KV Cache，导致大量计算浪费。传统 Prefix Caching 仅支持字节级精确前缀匹配，无法处理多文档 RAG 组合、文档顺序变更、对话历史增长等场景；且所有缓存操作与推理共享同一进程，造成 20%+ 推理吞吐损失。LMCache 采用解耦架构将缓存管理独立为单独进程，消除资源争用，实现零拷贝跨 GPU 共享和多层级并行加载，在 H200 + Qwen3-235B 上达到 14x TTFT 加速和 4x 解码加速。CacheBlend（EuroSys 2025 最佳论文）通过选择性重计算跨文档边界的关键 Token，解决了 Prefix Caching 的前缀匹配局限，使多文档查询提速 2-4x。

核心概念： 将 KV Cache 管理从推理引擎中解耦出来，是解决 Agent 推理成本爆炸和性能瓶颈的关键架构决策。

关键技术栈 / 关键要点：

1. KV Cache 二次方复杂度 — 注意力计算随输入长度二次方增长，128K Token 的 Agent 循环成本极高，单 GPU 日产约 15TB KV Cache
2. Prefix Caching 硬性上限 — 仅支持字节精确前缀匹配；Alibaba Cloud 数据显示 10% 的缓存块承载 77% 的命中，多数缓存因前缀规则无法复用
3. 进程内缓存性能税 — 缓存 I/O 与推理计算争抢同一 GPU 资源，Google TurboQuant 3-bit 量化虽无损但因进程内运行导致 20%+ 推理降速
4. LMCache 解耦架构 — 缓存管理独立进程 + 共享 GPU 内存，实现零资源争用、零拷贝跨 GPU 共享、多层级并行加载（GPU/CPU/SSD/云存储同时查）
5. CacheBlend 跨文档融合 — 识别跨文档边界的关键 Token 并选择性重计算，其余复用独立缓存；多文档查询提速 2-4x，RAG 场景下每个知识库文档成为可复用缓存资产
6. 经济可行性 — 仅需约 1% 命中率（每周复用 2-3 次）即可回本；1000 节点 10% 命中率可节省约 $29M/3年

关键结论 / 实践建议： 对于运行 Agentic 工作负载的自托管模型团队，KV Cache 管理不是未来优化而是当下成本结构决策；LMCache + CacheBlend 的解耦架构是目前最前沿的开源方案，兼容 vLLM/SGLang/TensorRT-LLM 及 NVIDIA/AMD GPU。

---

A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference by up to 14x.

Researchers at Stanford studied how AI agents actually spend their inference budgets. The finding that stood out is that about 62% of what gets sent to an agent on every call is just repeated content. The same system prompts, the same tool definitions, the same documents, fed in again and again.

So every time the agent takes a single step, you hand it everything from scratch. All the instructions, all the context, all the background. Even if it just processed the exact same information one second ago.

The expensive part isn't the thinking or the reasoning. It's the AI re-reading the same notes over and over.

The agent loop re-sending the same prompt payload (system prompt, tool definitions, documents) on every step,

Per-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from $30 to $0.40 per million tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context. So even though each token got cheaper, the total bill went up, since volume outran the price cuts.

Uber learned this the hard way. Rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in four months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.

The industry is optimizing the wrong variable. Making tokens cheaper doesn't help if most of those tokens shouldn't exist in the first place.

So, in this article, we're going to learn about a new open-source architecture that moves cache management out of the inference engine entirely. Teams running it are seeing up to 14x faster time-to-first-token, and most practitioners haven't caught up to it yet, so understanding it now puts you ahead of nearly everyone running inference today.

Diagram showing how LMCache works

What happens when you send a prompt #

Every time you hit a model with a prompt, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These K and V vectors capture how the model "understands" each token's relationship to every other token in the context.

This collection of K and V vectors is called the KV cache.

How the attention mechanism produces the KV cache

The computation scales quadratically with input length. Double the context and attention computation roughly quadruples. At 4K tokens (a simple chatbot turn) this is cheap. At 128K tokens (an agent loop with tools, documents, and history), it's expensive.

Quadratic compute cost curve across 4K, 32K, and 128K context lengths: double the context, quadruple the cost

A single MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.

The KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.

Think of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding. So you start over.

What prefix caching solves and where it stops #

The industry noticed this waste and built a feature called prompt caching to deal with it.

It works like this. If two consecutive requests share the same opening tokens (the "prefix"), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what's new.

The savings are real. Anthropic's implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.

But prefix caching has a hard ceiling.

The cached portion must be an exact, byte-for-byte prefix of the new request. Change anything in the cached region (even a single character) and you get a full cache miss.

This breaks in three common scenarios:

Three prefix caching scenarios: when it works and when it doesn't

• RAG with multiple documents: You cached document A alone and document B alone. A new query needs both. The second document's cached KV state is invalid because it was computed without awareness of the first document.
• Document order changes: The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.
• Growing conversation history: Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.

Alibaba Cloud's production data confirms the limitation. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.

Prefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning. Many real-world workloads don't look like that.

The caching performance tax #

There's a second problem with how caching works today, and it affects every KV cache tool available.

Every KV cache library runs inside the inference engine's process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources. They can't run at the same time.

When the engine is busy managing cache, it stops doing inference. When it's doing inference, cache operations wait.

Inference and cache I/O taking turns inside a single process

Think of it like a restaurant where the chef also has to run to the storage room to fetch ingredients between every dish. The cooking and the fetching can't happen simultaneously, so the kitchen slows down even though the chef is perfectly capable.

The impact is measurable. Google's TurboQuant, a recent KV cache quantization technique, compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.

The compression works perfectly, but the fact that it runs inside the same process as inference negates the benefit.

Cache management and inference serving are fundamentally different workloads. One is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).

Forcing both into the same process is like running a database and a web server in the same thread. It works until load hits, and then everything fights for the same resources.

LMCache and the disaggregated approach #

LMCache is an open-source project that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.

Going back to the restaurant analogy, this is like hiring a dedicated runner for the storage room. The chef never leaves the kitchen. The runner handles all the fetching and storing independently. Both work at full speed without waiting on each other.

Traditional single-process design vs LMCache's disaggregated architecture

In practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache "here are the block IDs I need" (tiny messages, almost no data).

All the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache's own process. The inference engine doesn't even notice it's happening.

This separation produces three concrete wins.

• No resource contention: Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.
• Zero-copy sharing across GPUs: In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.
• Multi-tier parallel loading: Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.

The four storage tiers (GPU memory, CPU RAM, local SSD, cloud storage) with sequential checking vs LMCache's parallel lookup across all tiers at once

The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.

And the economic math even works out even at very low cache hit rates. A cached prompt only needs to be reused two to three times per week (about 1% hit rate) for the system to pay for itself. At 10% hit rate on a 1,000-node deployment, the savings reach roughly $29M over three years.

LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.

Solving the prefix problem with CacheBlend #

LMCache's architecture solves the performance side of caching. But remember the prefix caching ceiling from earlier. You cached document A and document B separately, and now a query needs both. The second document's cache is broken because it was computed in isolation.

The LMCache team's research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.

The problem comes down to one thing. When you cache two documents independently, neither one "knows" about the other. So when you stitch their cached states together, the model can't jointly understand them. The connections between the two documents were never computed.

But it turns out that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.

CacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.

Prefix caching recomputing everything after the first document vs CacheBlend reusing all cached documents

The result is 2 to 4x faster processing for multi-document queries (the kind you see in RAG applications) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.

For teams building RAG systems, multi-document Q&A, or agents that accumulate context from multiple sources, this turns every document in the knowledge base into a reusable cached asset, regardless of what order it appears in or what other documents sit alongside it.

Built for production #

LMCache isn't a research prototype. It ships with the infrastructure that production teams expect.

• Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.
• Kubernetes operator for deployment
• CLI for debugging and benchmarking.

The fault tolerance design is worth noting.

If the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn't start cold.

If LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled but inference continues normally, and it reconnects automatically when the cache process recovers.

Neither failure takes the whole system down.

The bigger picture #

AI applications now generate roughly 15 TB of KV cache per GPU per day, and most of it gets thrown away. The infrastructure to manage, store, and reuse that data is being built right now.

For practitioners running agentic workloads on self-hosted models, KV cache management isn't a future optimization. It's a cost structure decision you're making today, whether you realize it or not.

LMCache is at the forefront of it. It's 100% open-source.

LMCache GitHub → (don't forget to star 🌟)

Those who like learning from videos, I will be doing a video deep dive on this architecture soon.

That's all for today, thanks for reading.

Cheers! :)