Inside auto routing: latency, cost & failover
How Gate picks a healthy provider in single-digit milliseconds and fails over without you noticing.
Health is a moving target
Provider health is not binary. A region degrades, a model queue backs up, a rate limit bites a single key. Gate keeps a rolling picture per provider, per model and per key: latency percentiles, error classes and throttle signals, updated from live traffic rather than synthetic probes.
The routing decision
When a request arrives for one of your aliases, Gate resolves the candidate providers, filters by your quota rules, and scores what remains on recent health. The decision costs single-digit milliseconds because everything it reads is already in memory on the routing path.
Failover is the same decision run again: when a provider errors mid-request, the retry lands on the next candidate with your original request intact. Most users learn a provider had an incident from the news, not from their error rates.