Profiling LLM Inference with SGLang and Torch Profiler
This blog is my attempt to learn how to profile an llm served using sglang and its in built torch profiler integration that comes out of box...
Topic
#CUDA-Graphs
2 篇笔记
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.