Rate limits & failover

Three things stand between a request and a model: the limits you set on your key, the capacity of the channels serving the model, and the health of those channels. The gateway absorbs most capacity and health problems itself, by queueing briefly and by retrying on another channel. What reaches you is a short, retryable 429 or 502.

Limits you set on a key#

Each key can have a Requests per minute limit and a Max concurrent requests limit. Set them under Limits & restrictions when you create or edit a key in API keys. Both are off by default.

A request that hits a limit waits up to 2 seconds for room. If none frees up, it fails with 429, the message This API key has reached its rate or concurrency limit., and a retry-after header in seconds. Use separate keys with their own limits to keep one workload from starving another. API keys covers the other key settings.

Capacity queueing#

Each channel can serve a limited number of requests at once. When the channel a request would go to is full, the gateway tries the other channels that serve the model. When all of them are full, the request joins a queue for the channel expected to free up first.

A request waits at most 10 seconds in total for capacity. If it is admitted, the response includes x-lovely-queued-ms with the time it waited. If not, it fails with 429, the message All upstream capacity for this model is busy. Retry shortly., and a retry-after header.

Queued requests are not charged until they run, and a request rejected with 429 is not charged at all.

Automatic failover#

When a channel fails, the gateway retries the request on another channel in the same channel group, up to three attempts per request. It fails over when a channel:

• times out, is unreachable, or returns 408, 409, 425, 429 or any 5xx;
• rejects the request for reasons on its side rather than yours;
• fails during a stream before the first token (see Streaming).

New conversations are spread across the channels that serve the model. A channel that returns more errors, or answers much more slowly than it usually does, gets a smaller share of new conversations until it recovers. A channel that keeps failing is taken out of rotation for a while, so later requests skip it. Conversations pinned to it move to a healthy channel.

Two cases are never retried:

• The request itself is invalid. When the model rejects it with 400, 404, 413 or 422, retrying elsewhere would fail the same way. You get that status with the reason.
• The stream has started. After the first token, the request is committed to its channel. A later failure arrives as an error event inside the stream.

Status codes#

The error type follows your API format: rate_limit_error (Anthropic), rate_limit_exceeded (OpenAI) or RESOURCE_EXHAUSTED (Gemini) for 429, and api_error (UNAVAILABLE in Gemini) for 502. The full list is in Errors.

Retry in your client#

The gateway already retries across channels, so a 429 or 502 means the problem lasted through several attempts. Retry it a few more times with exponential backoff and jitter, and honor retry-after when it is present.

The Anthropic and OpenAI SDKs do this by default, retrying 408, 409, 429 and 5xx twice. For batch jobs, raise the limit:

import os
from openai import OpenAI

client = OpenAI(base_url="https://lovelyrouter.com/v1", api_key=os.environ["LOVELY_API_KEY"], max_retries=5)

With plain HTTP, implement the same policy:

# --retry covers 429 and 5xx and honors retry-after
curl --retry 5 --retry-max-time 120 https://lovelyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $LOVELY_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "gpt-6.1-sol", "messages": [{"role": "user", "content": "Hi"}]}'

For streams, retry only when the error arrives before any content. If a stream breaks partway, you have already been charged for the partial output; decide whether to keep it or request the whole answer again.

Balance and spend limits#

Every request checks your balance and the key's spend limit before it runs. When the balance is at or below zero, or the key has spent its limit, the request fails with 402: Insufficient balance. Top up at the dashboard. or This API key has reached its spend limit. The error type is insufficient_quota in OpenAI format, billing_error in Anthropic format and RESOURCE_EXHAUSTED in Gemini format.

The check happens when a request starts, and charges settle a few seconds after it ends. A request that starts with a positive balance always runs to completion, so the balance can end slightly below zero; the next top-up covers it. See Credits & billing.