vLLM Semantic Router Releases d3-ultra: A 27B Open Multimodal Decision Model
vLLM Semantic Router released d3-ultra, a 26.09B Apache-2.0 multimodal decision model scoring 87.2 on its own SR-Text benchmark.
Topic
#vLLM
17 篇笔记
Kubernetes GPU Guide 2026: Scheduling, Sharing, Monitoring
Run GPUs on Kubernetes in 2026: install the NVIDIA GPU Operator, share GPUs with MIG or time-slicing, serve LLMs with vLLM, and monitor with DCGM.
GPU 共享怎么选：KAI vs HAMi vs MIG
Kubernetes 上的 GPU 共享：KAI、HAMi 与 MIG 对比。
KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
PaddleOCR-VL 服务问题排查与部署汇总
📝 摘要 本文汇总了 PaddleOCR-VL 服务在 vLLM 部署下的效果问题排查结论及完整部署方案。客户反馈 PaddleOCR-VL 丢失版面分析能力，根本原因是直接用 OpenAI SDK 裸调 vLLM 的 接口，只调用了 VLM 识别模型而绕过了版面分析模型 PP-DocLayoutV3。PaddleOCR-VL 的正确用法是两阶段 pipeline：版面分析 + VLM 识别。文…
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads comp
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
多模型路由进入 Agent 会话之后
模型路由在单轮问答中可以按难度分配成本，但进入 Agent 多轮会话后，还要处理上下文、工具状态、缓存和隐式状态。本文拆解会话级路由的智能与工程瓶颈，以及固定强模型、隔离子任务和有状态路由三条现实路线。
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
AI INFRA 全景分析与展望
AI INFRA 全景分析与展望 深度调研日期：2026-06-24 | 来源：多源 Web 搜索 + 对抗验证（24 个搜索代理，15+ 源站，70+ 可验证 claims） 📐 全景架构定位图 ---|------| | 核心创新 | PagedAttention — OS 式虚拟内存分页管理 KV Cache，将显存碎片率从 40% 降至 ⚠️ 注意：vLLM 上游无法提供 day-0 R…
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
vLLM + LMCache: A Starter Guide, No GPU Required
Get started easily: a single MacBook is all you need to develop vLLM + LMCacheFor New Contributors · Covering Frontend / L1 Eviction / L2 Storage / Observability If you ever skipped LMCache because you didn’t have a GPU on hand, this guide was written for you. LMCache’s multi-platform framework has already decoupled the GPU […]
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cach
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.