Efficient Memory Management for Large Language Model Serving with PagedAttention
Store each conversation's attention cache in small fixed-size blocks behind a table, the way an operating system pages memory, and two to four times as many tokens come off the same GPU.
Read the original ↗A language model generating text keeps a growing cache for every conversation it is serving, and in 2023 the serving systems stored each cache the way a beginner stores an array: one contiguous block, reserved at the maximum size up front. This paper measured how much of that memory held anything, found the answer was at best about four tenths, and fixed it with the oldest trick in the operating-systems textbook.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardInference at Scalewhy decoding is bound by memory, and paging the KV cache as one of the four levers that follow
- BoardThe Two Clockstime to first token and time per token as separate targets, which the paper folds into one averaged latency
- BoardPrefix Cachingthe shared start computed once across users, built on the paged blocks after this paper dropped the cache between turns
- BoardProcesses Threads and the Kernelvirtual memory and paging in full: page tables, frames and swap, the textbook the paper borrowed from
- BoardServing Engineswhere the GPU memory goes, block tables and copy-on-write, drawn from the operator's side
- LabThe batch size dialadd chats until the cache fills the card - the wall this paper moved by wasting less of it
- LabThe order slip railwait time that climbs gently and then runs away near capacity, the curve every serving result in the paper is plotted on
- VideoAttention in production: KV cache, MQA, GQA and MLAthe KV cache, and the attention variants that shrank each token's share of it after this paper took its size as fixed
- LessonInference at Scale (why an LLM is fast, and cheap)Uses the board Inference at Scale
- LessonThe Two Clocks (TTFT, TPOT, and why goodput is the number)Uses the board The Two Clocks
- LessonPrefix Caching (the same start, computed once)Uses the board Prefix Caching
- LessonContainer 01: Fork and ExecUses the board Processes Threads and the Kernel
- LessonContainer 02: UTS and PID NamespacesUses the board Processes Threads and the Kernel
- LessonContainer 04: /proc and /devUses the board Processes Threads and the Kernel
- LessonProcesses, Threads & the KernelUses the board Processes Threads and the Kernel
- LessonThe address book every process carriesUses the board Processes Threads and the Kernel
- LessonWhose turn, and who decidesUses the board Processes Threads and the Kernel
- LessonThe program that starts programsUses the board Processes Threads and the Kernel
- LessonAI/ML Ops - the deep-dive setUses the board Serving Engines
Further reading
Read the original for the full argument and its evidence.
- The paper (arXiv) ↗Published at SOSP 2023. The memory-waste figure on page 2 and the block-table walkthrough in section 4.3 are the argument; section 8 says where the idea does not apply.
- vLLM (GitHub) ↗The open-source engine the paper describes, which has grown far past the 8,500 lines of Python and 2,000 of C++ and CUDA the paper reports.
- The launch post, 20 June 2023 ↗The vLLM team's announcement, three months before the paper, comparing throughput against Hugging Face Transformers and Text Generation Inference rather than the paper's Orca and FasterTransformer.