Training gets the glory; serving gets the users. How efficiently you can run a model for many people at once — throughput — decides your cost per request and how many customers a GPU can support. The central technique is batching.
Why batching matters
A GPU is a massively parallel machine. Running one request at a time barely uses it — like driving a bus with one passenger. Batching combines many requests and processes them together, using the hardware far more fully. More requests per batch means more throughput per GPU, which means lower cost per request. Serving efficiency is largely the art of keeping the batch full.
One request wastes the GPU. A full batch uses it. Throughput is mostly about never letting the bus leave half-empty.
The hard part: variable-length generation
LLM serving has a twist that makes batching tricky: different requests generate different amounts of text and finish at different times. Naive batching wastes capacity waiting for the slowest request. Modern serving systems use continuous batching — as soon as one request finishes, a new one takes its slot, keeping the GPU busy. Combined with efficient memory management for the KV cache, this dramatically raises how many concurrent users a GPU can serve.
Why it's worth understanding
The gap between a naive and a well-tuned serving setup can be several-fold in cost — the same model, the same hardware, but many more users served per dollar. This is why inference-serving frameworks are their own field now. For anyone deploying LLMs at scale, throughput engineering — batching, KV-cache management, scheduling — is where a huge amount of the real cost savings live, quietly, behind every fast and affordable AI product.
