KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
Topic
#KV-cache
20 篇笔记
AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
Agent 终章（Harness 成本篇）：一次百炼账单降低 88% 实战
关于优化 token，本文或许能够提供一些经验和参考
Why model routing must be in the harness
Inference spend is rising as agents are deployed more widely, take on more complex tasks, and run longer autonomously. Enterprises are start...
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads comp
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
多模型路由进入 Agent 会话之后
模型路由在单轮问答中可以按难度分配成本，但进入 Agent 多轮会话后，还要处理上下文、工具状态、缓存和隐式状态。本文拆解会话级路由的智能与工程瓶颈，以及固定强模型、隔离子任务和有状态路由三条现实路线。
22580: From GPT2 to Kimi3, Explained
Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside KimiK3 (2026). We scaled up by a factor of 22,58...
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Your KV Caching Is Broken
A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference b...
Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction
📝 摘要 Google Research 发布了一种将 Multi-Token Prediction (MTP) 增层改造（retrofit）到已冻结的 Gemini Nano v3 生产模型上的方法，在不修改基座模型权重的前提下，大幅加速 Pixel 设备上的端侧 LLM 推理。该方法已在 Pixel 9 和 10 系列实机部署，使 AI Notification Summaries、Proo…
AI INFRA 全景分析与展望
AI INFRA 全景分析与展望 深度调研日期：2026-06-24 | 来源：多源 Web 搜索 + 对抗验证（24 个搜索代理，15+ 源站，70+ 可验证 claims） 📐 全景架构定位图 ---|------| | 核心创新 | PagedAttention — OS 式虚拟内存分页管理 KV Cache，将显存碎片率从 40% 降至 ⚠️ 注意：vLLM 上游无法提供 day-0 R…
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
vLLM + LMCache: A Starter Guide, No GPU Required
Get started easily: a single MacBook is all you need to develop vLLM + LMCacheFor New Contributors · Covering Frontend / L1 Eviction / L2 Storage / Observability If you ever skipped LMCache because you didn’t have a GPU on hand, this guide was written for you. LMCache’s multi-platform framework has already decoupled the GPU […]
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cach
KV Caching in LLMs, Clearly Explained
You must have seen it every time you use ChatGPT or Claude that the first token takes noticeably longer to appear. Then the rest stream out ...
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.