A powerful pattern has emerged for squeezing better answers out of existing models: don't rely on one, orchestrate several. "Mixture-of-agents" runs multiple models (or multiple calls) that propose, critique, and refine, and the combined result often beats any single model working alone.
The idea
Instead of one model answering once, you set up a small collaboration: several models generate candidate answers, then one or more models critique and synthesize them into a better final response. It's like a panel of experts drafting independently, then a chair combining the best of each — the diversity of perspectives catches errors and surfaces better ideas than any lone attempt.
One model gives one perspective. Several models, cross-checking each other, give something closer to a considered consensus.
Why it works
Different models (or the same model sampled differently) make different mistakes and have different strengths. Aggregating them cancels out individual errors and combines strengths — the same logic behind self-consistency and ensembles, applied at the level of whole agents. A critique-and-refine loop adds another lever: models are often better at spotting flaws than avoiding them, so having one model review another's work improves quality.
The cost tradeoff
This isn't free — running several models and critique passes multiplies compute and latency. So it fits high-value tasks where quality justifies the cost, not high-volume cheap requests. It's a form of test-time compute: spend more inference to get a better answer.
Where it fits
Mixture-of-agents shines for hard problems, high-stakes outputs, and cases where a single model plateaus. It connects to broader trends — Sakana's orchestration approach, multi-agent systems, ensembling — all exploiting the same insight: coordination can beat raw capacity. When one model isn't enough, before reaching for a bigger model, consider whether a well-orchestrated panel of models gets you there — often it does, and cheaper.
