AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
Topic
#speculative-decoding
10 篇笔记
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction
📝 摘要 Google Research 发布了一种将 Multi-Token Prediction (MTP) 增层改造（retrofit）到已冻结的 Gemini Nano v3 生产模型上的方法，在不修改基座模型权重的前提下，大幅加速 Pixel 设备上的端侧 LLM 推理。该方法已在 Pixel 9 和 10 系列实机部署，使 AI Notification Summaries、Proo…
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.
Editing Files at 1000 Tokens per Second
A new model and inference method for high-accuracy full-file edits at 1000 tokens/s.