Prompt caching

Prompt caching lets a model reuse the work it did on a prompt prefix it has seen recently. Cached input costs a fraction of the normal input price: on claude-sonnet-5.5, a cache read costs $0.10 per million tokens against $2.00 for uncached input. For agents and long chats, where every turn resends the whole history, most input tokens can be served from the cache.

Two things make it work through Lovely Router:

1. Vendor caching. You use each vendor's caching exactly as documented. Request fields such as cache_control and prompt_cache_key pass through unchanged.
2. Conversation affinity. A cache lives on the channel that wrote it. The gateway recognizes the turns of one conversation and sends them back to the same channel, so the cache is there when the next turn arrives.

Anthropic#

Claude caches only what you mark. Put cache_control on the last block of the stable prefix (tools, system prompt, earlier turns). You can set up to four breakpoints, or put one cache_control at the top level of the request to cache everything up to the last block.

response = client.messages.create(
    model="claude-sonnet-5.5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT_AND_REFERENCE_DOCS,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=history,
)
print(response.usage)

The response reports cache_creation_input_tokens (written, billed at the cache-write price) and cache_read_input_tokens (read, billed at the cache-read price). Prefixes below the model's minimum length are not cached. Anthropic's caching guide covers placement and what invalidates a cache.

OpenAI#

GPT models cache long prompt prefixes automatically; there is nothing to mark. Keep the stable part of the prompt first and the changing part last. Setting prompt_cache_key to a stable value per conversation or per user raises the hit rate, and the gateway also uses it to keep that conversation on one channel.

response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=history,
    prompt_cache_key="conv_8f2a91",
)
print(response.usage.prompt_tokens_details.cached_tokens)

usage.prompt_tokens_details.cached_tokens is billed at the cache-read price and the rest of prompt_tokens at the input price. GPT models have no separate cache-write charge. See OpenAI's caching guide.

Gemini#

Gemini models cache repeated prefixes on their own (implicit caching). Cached tokens appear in usageMetadata.cachedContentTokenCount and are billed at the cache-read price, with no cache-write charge. Explicit context caching through the cachedContents API is not available.

How conversations are recognized#

The gateway gives each conversation a fingerprint. It uses the first match from this list:

The last rule needs nothing from you. Later turns of a conversation only append messages, so the system prompt and the first user message stay the same and every turn gets the same fingerprint. Only the first 4,000 characters of each are compared.

Fingerprints are scoped to your API key, the model and the channel group. The same conversation sent with a different key or to a different model starts a new pin.

How long a conversation stays pinned#

The first successful call of a conversation pins it to the channel that served it. Later turns go to that channel first.

• Duration. A pin lasts up to one hour and is renewed while the conversation is active. After 30 to 60 minutes without a request, it expires and the next turn is placed like a new conversation.
• Expired cache. In the Anthropic Messages format, a pin also ends when the conversation has been idle longer than its prompt cache lives: 5 minutes, or 1 hour when your cache_control breakpoints set ttl: "1h". The cache it protected is gone by then, so the next turn is placed like a new conversation instead of waiting on the channel that served last.
• Busy channel. If the pinned channel is at capacity, the turn waits for it briefly (about one second for short prompts, up to ten seconds for long ones) rather than moving and paying for a new cache write. If it is still full after that, the turn moves to another channel.
• Failed channel. If the pinned channel is failing, the turn moves to a healthy channel and the pin follows it. That turn writes a fresh cache; the turns after it read from it again.

Send an explicit session id#

The automatic fingerprint fails when the first user message changes between turns, for example when your app rewrites or summarizes old history. In that case, send a stable id per conversation. A header works in every API format:

curl https://lovelyrouter.com/v1/messages \
  -H "x-api-key: $LOVELY_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-session-id: conv_8f2a91" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-5.5", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Continue the report."}]}'

Body fields work too: metadata.user_id in Anthropic format, prompt_cache_key in OpenAI format. Use one id per conversation; reusing one id across many unrelated conversations puts them all on one channel.

See cache use in logs and billing#

Logs show, for each call:

The dashboard overview shows your cache hit rate across all calls. Each kind of token is billed at its own price; see How a call is charged.