Kubernetes GPU Guide 2026: Scheduling, Sharing, Monitoring
Run GPUs on Kubernetes in 2026: install the NVIDIA GPU Operator, share GPUs with MIG or time-slicing, serve LLMs with vLLM, and monitor with DCGM.
Topic
#gpu
14 篇笔记
GPU 共享怎么选：KAI vs HAMi vs MIG
Kubernetes 上的 GPU 共享：KAI、HAMi 与 MIG 对比。
AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
Profiling LLM Inference with SGLang and Torch Profiler
This blog is my attempt to learn how to profile an llm served using sglang and its in built torch profiler integration that comes out of box...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
前OpenAI推理负责人JT与Gemini pretrain负责人Rohan: Continue learning的未来
📝 摘要 本文是红杉资本对 Core Automation 联创 Jerry Tworek（前 OpenAI o1 reasoning 负责人）和 Rohan Anil（前 Google Gemini pretrain 负责人）的访谈。两人共同阐述了 Transformer 架构的根本局限——训练停止后无法真正持续学习——以及 Core Automation 寻找"Transformer 替代者…
nvml-mock
nvml-mock 📝 摘要 nvml-mock 是 NVIDIA 官方的 Kubernetes Helm Chart，通过 CGo 编译的 mock 共享库，在无物理 GPU 的集群中模拟完整的多 GPU 环境，使 、device plugin、DRA driver 等 GPU 相关 K8s 组件无需真实硬件即可运行和测试。它部署为 DaemonSet + ConfigMap + RBAC，在…
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Your KV Caching Is Broken
A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference b...
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
SkyPilot Sandboxes: Run Agent Code on Your Own Kubernetes, at Scale
SkyPilot runs untrusted, LLM-generated code in sandboxes on the Kubernetes clusters you already own. A single cluster sustains 50,000+ sandboxes, with multi-cluster support to go further. Individual launches in less than a second. All at up to one-tenth the cost of hosted alternatives, with your code and data never leaving your cloud.