With dozens of capable models available, "which LLM should I use?" is one of the most common — and most mishandled — questions in applied AI. The mistake is picking the benchmark leader. The right answer depends on your specific needs.
The dimensions that actually matter
- Quality on your task — not a generic benchmark, but your real cases. Test candidates on your own examples.
- Cost — price per token, multiplied by your expected volume. At scale, this dominates.
- Latency — how fast does it respond? Critical for chat and voice, less so for background jobs.
- Privacy & control — can you send data to a third-party API, or do you need to self-host? Regulated industries often need the latter.
- Context length — does your use case need to feed in long documents?
- Reliability & maturity — rate limits, uptime, and how stable the provider is.
The best model for your product is rarely the smartest one. It's the one that clears your quality bar at your budget, speed, and privacy needs.
A practical process
- Define your bar — what quality is "good enough" for your task?
- Build a small eval set from real examples.
- Test a shortlist — a frontier model, a strong mid-tier, and a cheap/open option.
- Find the cheapest that clears the bar — don't overpay for quality you don't need.
- Consider a cascade — a cheap model for common cases, escalating hard ones to a stronger model.
The mindset
Model choice is engineering, not fandom. The frontier model is for the hard minority of tasks; a cheaper or open model handles the rest at a fraction of the cost. The teams that win aren't the ones always calling the biggest model — they're the ones who matched the model to the job. Start with your requirements, not the leaderboard.
Was this useful?
