The eye-watering numbers in AI headlines are usually training costs — millions of dollars to train a frontier model. But for a successful product, those are often not the biggest expense. Inference — running the model to serve users, over and over — is the cost that never stops.
One-time vs forever
Training is a large, one-time capital cost. You pay it once, get a model, and it's done. Inference is an operating cost that scales with every user, every request, every token generated. A model used by millions of people generates inference costs continuously, and those add up relentlessly. Over a popular product's life, total inference spend routinely exceeds the training bill — often by a wide margin.
Training is buying the factory. Inference is the electricity bill — and it arrives every month, forever.
Why this reframes engineering
Once you internalize that inference dominates at scale, a lot of the field's obsessions make sense. Quantization, speculative decoding, distillation into smaller models, prompt caching, right-sizing, cascades — all of it targets the cost of serving, because a small per-request saving multiplied by billions of requests is enormous. A 2x inference speedup can be worth more than a smarter model.
The strategic takeaway
For anyone building an AI product, the lesson is to design for inference economics from the start: use the smallest model that clears your quality bar, cache aggressively, route easy requests to cheap models, and measure cost per request as a first-class metric. The teams that win aren't always the ones with the best model — they're the ones who serve a good-enough model most efficiently. Training gets the headlines; inference pays the bills.
