Your Inference Server is Secretly a Learner: Open-Sourcing Reef for Continual Self-Improving Agents
1. Before all the “RSI” hype becomes realBefore today's excitement around RSI is fully realized, we want to open-source Reef, an infrastruct...
Topic
#ai-infra
21 篇笔记
GPU 共享怎么选：KAI vs HAMi vs MIG
Kubernetes 上的 GPU 共享：KAI、HAMi 与 MIG 对比。
AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
nvml-mock
nvml-mock 📝 摘要 nvml-mock 是 NVIDIA 官方的 Kubernetes Helm Chart，通过 CGo 编译的 mock 共享库，在无物理 GPU 的集群中模拟完整的多 GPU 环境，使 、device plugin、DRA driver 等 GPU 相关 K8s 组件无需真实硬件即可运行和测试。它部署为 DaemonSet + ConfigMap + RBAC，在…
Agency: Secure, scalable sandboxes for agents
This is part of our series on how we're AI-pilling Sierra.In our last few posts, we talked about what happens when every employee has a sing...
Why Kubernetes is the wrong tool for agentic workloads
At some point, everyone building ai infra wonders whether they should use Kubernetes. There’s no straight answer for it because it depends o...
Your KV Caching Is Broken
A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference b...
Volcano 调度框架深度解读
Volcano 调度框架深度解读 调研日期: 2026-07-01 | 版本: v1.15.0 (2026-06) | 置信度: 全部 findings 经 3-vote 对抗验证，均 High confidence 官方站点: volcano.sh | GitHub: volcano-sh/volcano 一、项目概览与 CNCF 状态 Volcano 是 CNCF 孵化项目（2022-03-2…
AI INFRA 全景分析与展望
AI INFRA 全景分析与展望 深度调研日期：2026-06-24 | 来源：多源 Web 搜索 + 对抗验证（24 个搜索代理，15+ 源站，70+ 可验证 claims） 📐 全景架构定位图 ---|------| | 核心创新 | PagedAttention — OS 式虚拟内存分页管理 KV Cache，将显存碎片率从 40% 降至 ⚠️ 注意：vLLM 上游无法提供 day-0 R…
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
vLLM + LMCache: A Starter Guide, No GPU Required
Get started easily: a single MacBook is all you need to develop vLLM + LMCacheFor New Contributors · Covering Frontend / L1 Eviction / L2 Storage / Observability If you ever skipped LMCache because you didn’t have a GPU on hand, this guide was written for you. LMCache’s multi-platform framework has already decoupled the GPU […]
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway
LiteLLM is moving its AI gateway to Rust: 15x throughput, 11x less memory, and sub-1ms per-request overhead. No v2, no migration, your config stays the same.
SkyPilot Sandboxes: Run Agent Code on Your Own Kubernetes, at Scale
SkyPilot runs untrusted, LLM-generated code in sandboxes on the Kubernetes clusters you already own. A single cluster sustains 50,000+ sandboxes, with multi-cluster support to go further. Individual launches in less than a second. All at up to one-tenth the cost of hosted alternatives, with your code and data never leaving your cloud.
Building cloud agent infrastructure: what's different, and what we learned
Most agent frameworks today assume a desktop. One user, one machine, one process. The agent runs while the laptop is open, writes to a local...
How to Build AI Agent Swarms (Complete Guide)
This is a complete A–Z breakdown of AI Agent Swarms what they are, how to use them.Why they change everything about how you work with AI.B...
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cach
Kubernetes AI/ML 基础设施项目全景调研
Kubernetes AI/ML 基础设施项目全景调研 调研日期：2026-05-08 | 覆盖项目：Volcano、HAMi、Ray、LeaderWorkerSet、Kueue、KServe、Kubeflow、Seldon Core、vLLM、Triton、GPU Operator 等 二、核心项目热度总览 | 项目 | GitHub ⭐ | CNCF 状态 | 最新版本 | 核心定位 | |-…