Latency and Performance

OpenRouter is designed with performance as a top priority. OpenRouter is heavily optimized to add as little latency as possible to your requests.

Minimal Overhead

OpenRouter is designed to add minimal latency to your requests. This is achieved through:

Edge computing using Cloudflare Workers to stay as close as possible to your application
Efficient caching of user and API key data at the edge
Optimized routing logic that minimizes processing time

Performance Considerations

Cache Warming

When OpenRouter’s edge caches are cold (typically during the first 1-2 minutes of operation in a new region), you may experience slightly higher latency as the caches warm up. This normalizes once the caches are populated.

Credit Balance Checks

To maintain accurate billing and prevent overages, OpenRouter performs additional database checks when:

A user’s credit balance is low (single digit dollars)
An API key is approaching its configured credit limit

OpenRouter expires caches more aggressively under these conditions to ensure proper billing, which increases latency until additional credits are added.

Model Fallback

When using model routing or provider routing, if the primary model or provider fails, OpenRouter will automatically try the next option. A failed initial completion unsurprisingly adds latency to the specific request. OpenRouter tracks provider failures, and will attempt to intelligently route around unavailable providers so that this latency is not incurred on every request.

The Latency & DX Optimization Cookbook

When building interactive developer tools, coding agents, or low-latency conversational apps, total perceived latency is governed by two separate phases:

Time to First Token (TTFT): Network transit + provider queue wait time + prompt prefill.
Token Throughput (TPS): Decode and generation streaming speed.

Total Latency = TTFT (Network + Queue + Prefill) + (Output Tokens / Generation TPS)

Below are drop-in request configurations for common developer experience (DX) optimization scenarios. See the provider routing guide for the full API reference on sort, partition, preferred_max_latency, and preferred_min_throughput.

Recipe 1: Bounding Peak-Hour Queue Latency

Scenario: During peak global traffic hours (e.g. US morning rush), popular open-weight inference hosts can experience queue congestion, causing Time to First Token (TTFT) to spike. Solution: Use preferred_max_latency with a percentile cutoff (such as p90).

{
  "model": "deepseek/deepseek-chat",
  "messages": [{ "role": "user", "content": "Explain continuous batching in 3 sentences." }],
  "provider": {
    "preferred_max_latency": {
      "p90": 2.5
    }
  }
}

Rolling 5-Minute Window: Evaluates performance over the last 5 minutes, rapidly adapting when a host starts queueing.
Soft Reordering (Zero 404 Risk): Providers that meet the threshold are promoted to the front of the candidate list. If all providers are experiencing high load, the request still executes on the best available host rather than failing closed.

Recipe 2: The “Fastest Provider on a Budget”

Scenario: You want to minimize token costs without suffering through painfully slow streaming speeds (under 15 tokens/sec). Solution: Combine sort: "price" with preferred_min_throughput.

{
  "model": "meta-llama/llama-3.3-70b-instruct",
  "messages": [{ "role": "user", "content": "Refactor this function to be async." }],
  "provider": {
    "sort": {
      "by": "price"
    },
    "preferred_min_throughput": {
      "p90": 40
    }
  }
}

OpenRouter filters for hosts that have sustained at least 40 tokens/second for 90% of requests over the last 5 minutes, and routes to the cheapest provider within that high-throughput group.

Recipe 3: Autonomous Agent Loops & Large Contexts

Scenario: Autonomous agent harnesses send 20,000 to 90,000+ input tokens per turn. Re-computing attention over large contexts without a warm cache adds significant prefill delay on every step. Solution: Preserve sticky routing and provider KV caching across turns.

{
  "model": "anthropic/claude-3.7-sonnet",
  "session_id": "workspace-session-984a-4e21",
  "messages": [
    { "role": "system", "content": "You are a helpful coding assistant with access to local tools." },
    { "role": "user", "content": "Run test suite and fix failing cases." }
  ]
}

Golden Rules for Agent Caching:

Pass a Consistent session_id: OpenRouter stores a 10-minute best-effort pin directing follow-up turns back to the exact provider endpoint holding the warm KV cache.
Keep the Prefix Static: Ensure system prompts, repository maps, and tool definitions appear at the beginning of the prompt and remain byte-identical across turns. Dynamic timestamps or session metadata should be placed at the end.
Keep the Model Slug Identical: Changing from claude-3.7-sonnet to claude-3.7-sonnet:nitro mid-conversation invalidates the sticky key.
Avoid Hardcoded provider.order: Setting explicit provider.order disables sticky-session reordering and load balancing.

Recipe 4: Multi-Model Latency Flattening

Scenario: You have a fallback list of several interchangeable models (e.g. Llama 3.3 70B, Mistral Large, Claude 3.5 Haiku). By default, OpenRouter tries all endpoints of Model A before attempting Model B, even if Model B is currently idle and faster. Solution: Flatten the fallback grouping by setting partition: "none".

{
  "models": [
    "anthropic/claude-3.5-haiku",
    "meta-llama/llama-3.3-70b-instruct",
    "google/gemini-2.5-flash"
  ],
  "messages": [{ "role": "user", "content": "Format this JSON object." }],
  "provider": {
    "sort": {
      "by": "throughput",
      "partition": "none"
    }
  }
}

Setting partition: "none" pools all endpoints across all listed models and routes directly to whichever endpoint has the highest measured throughput right now.

Quick Reference Summary

Goal	Recommended Configuration	Latency Impact
Bound Worst-Case Peak Delays	`preferred_max_latency: { p90: 2.5 }`	Evades congested queues; evaluates 5-min rolling percentiles.
Fastest Generation on a Budget	`sort: "price"` + `preferred_min_throughput: { p90: 40 }`	Prevents routing to oversubscribed slow commodity hosts.
Agent Loops / Long Contexts	Stable `session_id` + static prompt prefix	Substantially reduces prefill time and cost via provider KV caching.
Global Speed Across Models	`models: [...]` + `sort: { by: "throughput", partition: "none" }`	Routes to fastest available endpoint across all candidate models.

Overview

Models & Routing

Tool Calling

Features

Ori

Privacy

Best Practices

Community

Minimal Overhead

Performance Considerations

Cache Warming

Credit Balance Checks

Model Fallback

The Latency & DX Optimization Cookbook

Recipe 1: Bounding Peak-Hour Queue Latency

Recipe 2: The “Fastest Provider on a Budget”

Recipe 3: Autonomous Agent Loops & Large Contexts

Recipe 4: Multi-Model Latency Flattening

Quick Reference Summary

​Minimal Overhead

​Performance Considerations

​Cache Warming

​Credit Balance Checks

​Model Fallback

​The Latency & DX Optimization Cookbook

​Recipe 1: Bounding Peak-Hour Queue Latency

​Recipe 2: The “Fastest Provider on a Budget”

​Recipe 3: Autonomous Agent Loops & Large Contexts

​Recipe 4: Multi-Model Latency Flattening

​Quick Reference Summary

Minimal Overhead

Performance Considerations

Cache Warming

Credit Balance Checks

Model Fallback

The Latency & DX Optimization Cookbook

Recipe 1: Bounding Peak-Hour Queue Latency

Recipe 2: The “Fastest Provider on a Budget”

Recipe 3: Autonomous Agent Loops & Large Contexts

Recipe 4: Multi-Model Latency Flattening

Quick Reference Summary