The architecture behind our multi-supplier gateway: health checks, circuit breakers, standby warm pools, and degradation policies.
99.99% uptime on top of providers that individually deliver around 99.9% is not a marketing number — it is what you get from independent failure domains, aggressive health checks, and a routing layer that decides in single-digit milliseconds.
Health checks
Every upstream endpoint is probed every 2 seconds with a synthetic request that exercises the real inference path, not a status page. We track p50 and p99 latency, error rate, and structural validity of the response. A provider is marked degraded before it starts returning errors — latency drift is the earliest signal.
Circuit breakers
- Closed: normal routing, all traffic flows
- Half-open: 5% of traffic probes recovery while the rest routes to standby
- Open: no traffic; reopened only after 30 consecutive healthy probes
Warm standby pools
Cold failover costs you the first-token latency of a fresh connection plus any provider-side cold start. We keep persistent connections open to every standby and replay the in-flight request against the new target, so the client sees a single slower response rather than an error.
The best failover is the one your users never notice. Median added latency from the routing layer is under 4ms; a full failover event adds 180-400ms to the affected request.
Degradation policy
When no equivalent model is healthy, you choose what happens: fall back to a cheaper tier, serve a stale cache entry, or fail fast. Defaulting to fail-fast is the honest choice for agentic workloads where a worse answer is more expensive than no answer.