For most of the modern AI era, "run the model" meant "call a server." That assumption is breaking. Capable models now run entirely on consumer hardware — phones, laptops, edge devices — and the shift changes more than latency.
Three reasons it matters
Privacy. A model that runs on your device can answer without sending your data anywhere. For messages, documents, and health data, that's not a nicety — it's often the difference between a feature being acceptable and being a non-starter.
Cost. On-device inference has no per-call price. For features used millions of times a day, moving the common cases off the server changes the unit economics entirely.
Latency and offline. Local inference answers in milliseconds and keeps working with no connection. Interactions that felt sluggish over the network feel instant on-device.
The cloud isn't going away. But the default is shifting: do it locally if you can, reach for the server only when you must.
What makes it possible
Two forces converged: models got dramatically more capable at small sizes (via distillation and better data), and consumer hardware picked up dedicated neural accelerators. Together they moved the line of "what's possible locally" far enough that everyday assistant features now live on the device.
The architecture that follows
The mature pattern is hybrid: a small on-device model handles the frequent, latency-sensitive, privacy-sensitive work, and hands off to a larger cloud model only for the hard minority. Designing that handoff well — deciding what stays local and what escalates — is becoming a core product skill, not just an infrastructure detail.
