Guides

Rate Limiting

Per-key rate limits protect against burst traffic and runaway token consumption. When a limit is hit, the gateway returns HTTP 429 with headers that tell you exactly when to retry.

Limit types

RPM — Requests per minute

Counts every request in a fixed 60-second window. Resets every minute. Good for protecting against burst spikes from a single client.

TPD — Tokens per day

Cumulative prompt + completion tokens across all requests. Resets at midnight UTC. Good for capping total daily cost for a key.

Limits are configured per API key at the platform level. Contact support@inferexai.in to set or adjust limits for your keys. If no limit is configured, the key has no rate limit.

429 response — RPM

text

HTTP/1.1 429 Too Many Requests
Retry-After: 42
X-RateLimit-Limit-RPM: 60
Content-Type: application/json

{"detail": "rate limit exceeded — too many requests per minute"}

Retry-After — seconds until the current window expires. Sleep this long before retrying.
X-RateLimit-Limit-RPM — the configured RPM limit for this key.

429 response — TPD

text

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit-TPD: 500000
X-RateLimit-Used-TPD: 500000
Content-Type: application/json

{"detail": "daily token limit exceeded — resets at midnight UTC"}

X-RateLimit-Limit-TPD — the daily token limit.
X-RateLimit-Used-TPD — tokens consumed today. Resets at midnight UTC — no Retry-After header is set.

Handling 429 in Python

rate_limit.py

import time
from inferexai import InferexAI, InferexAIError

client = InferexAI(api_key="sk-live-your-key")

def chat_with_rate_limit(messages, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model="default",
                messages=messages,
            )
        except InferexAIError as e:
            if e.status != 429:
                raise
            retry_after = int(e.response.headers.get("Retry-After", "10"))
            print(f"Rate limited. Retrying in {retry_after}s...")
            time.sleep(retry_after)
    raise RuntimeError("Rate limit retries exhausted")

Handling 429 in Node.js

rate_limit.ts

import InferexAI from "inferexai";

const client = new InferexAI({ apiKey: "sk-live-your-key" });

async function chatWithRateLimit(messages, maxRetries = 5) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await client.chat.completions.create({
        model: "default",
        messages,
      });
    } catch (err) {
      if (err?.status !== 429) throw err;
      const retryAfter = parseInt(err.headers?.["retry-after"] ?? "10", 10);
      console.log(`Rate limited. Retrying in ${retryAfter}s...`);
      await new Promise(r => setTimeout(r, retryAfter * 1000));
    }
  }
  throw new Error("Rate limit retries exhausted");
}

Tips

✓Always read the Retry-After header rather than using a fixed backoff — it tells you exactly how long to wait.

✓For voice agents and real-time applications, set RPM conservatively and use streaming to reduce perceived latency.

✓TPD limits reset at midnight UTC, not midnight in your local timezone.

✓Create separate keys per application. If one app hits its limit, other keys are unaffected.

✓Monitor per-key token usage in Dashboard → Observability to tune limits before they bite in production.