Latency, cost, context length, tool-call reliability — we benchmark the big three across 6 real-world scenarios.
We ran the three leading frontier families through six production-shaped scenarios on identical prompts, same temperature, same day, routed through the same gateway. Here is what actually separates them.
The scenarios
- Multi-file code refactor with 180k tokens of context
- Structured extraction from messy invoices into a strict JSON schema
- Customer support reply with tone constraints and a tool call
- Long-document summarization at 600k tokens
- Chain-of-tool-calls agent loop, eight steps deep
- Creative long-form drafting with a style guide
What we found
Tool-call reliability is the widest gap. On the eight-step agent loop, the best model completed 94% of runs without a malformed call; the worst was at 71%. On plain single-turn generation the three families are within a few points of each other and the choice should come down to price and latency.
If your workload is one prompt in, one answer out, pick on cost. If it is agentic, pick on tool-call reliability and accept the price — retries are more expensive than the better model.
A default routing policy
{
"routes": [
{ "match": "tools:true", "model": "claude-sonnet-4-5" },
{ "match": "context>400000", "model": "gemini-2.5-pro" },
{ "match": "latency<300ms", "model": "qwen-3.8-flash" },
{ "default": "gpt-5" }
]
}
Start there, then let the gateway's per-route quality metrics tell you where to deviate. Re-evaluate quarterly — the ordering has changed twice in the last year.