API Rate Limited in Production at 3 AM: Building Retry Logic That Actually Works

The 3 AM Page

It is 3:14 AM. Your phone buzzes. The summary says: "Order processing pipeline backed up — 8,400 jobs queued, error rate 47%."

You stumble to your laptop. The Sentry dashboard is full of 429 Too Many Requests from the Stripe webhook handler. You look closer — the actual webhook traffic is normal, but your retry logic, written 18 months ago by someone who has since left, is hammering Stripe with retries every 100ms whenever it sees a 429. Stripe is now rate-limiting your IP entirely.

You manually push a fix. By 4:30 AM the queue drains. By 9 AM your manager wants to know why a third-party rate limit took down your service.

This guide is the answer. It covers the actual mechanics of rate limits, how to read 429 responses correctly, the retry math that survives production load, and the patterns — circuit breakers, queues, idempotency — that make a system resilient instead of fragile.

For broader context on shipping reliable systems, our TypeScript best practices for 2026 covers type-safe error handling patterns, and our API testing tools roundup for 2026 lists the tools to validate retry behavior in CI.

How Rate Limits Actually Work

Three algorithms power 99% of production rate limiters. Understanding which your provider uses determines how to handle their 429s.

Token Bucket

A bucket holds N tokens. Each request consumes one. The bucket refills at a fixed rate (e.g., 100 tokens per second). When empty, requests fail with 429.

Used by: AWS APIs, OpenAI, most cloud APIs. Allows bursts up to bucket size, smooths to refill rate over time.

Sliding Window

Tracks request count in a moving time window (e.g., last 60 seconds). When count exceeds limit, requests fail.

Used by: GitHub API (5000 requests per hour, sliding), Twitter/X v2 API. More accurate than fixed window but more expensive to compute server-side.

Fixed Window

Counts requests in discrete time windows (e.g., 100 per minute, resetting at the top of each minute). When count exceeds limit, requests fail until the window resets.

Used by: many internal services and budget APIs. Has a "boundary effect" — doubles allowed in transit during the brief moments around reset boundaries.

Concurrency Limits (Less Common)

Limits the number of in-flight requests rather than rate. Twilio uses this for some endpoints, as do some database APIs.

The implementation matters because it changes how you should retry. With a token bucket, waiting briefly often refills enough to succeed. With sliding window, you may need to wait for the entire window to roll forward.

Reading 429 Responses Correctly

The HTTP standard for 429 is RFC 6585, and the relevant retry header is in RFC 7231. In practice, providers add their own headers.

Retry-After

The standard. Two valid forms:

• Seconds: Retry-After: 30
• HTTP date: Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Code that handles both:

function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;
  const seconds = Number(header);
  if (!Number.isNaN(seconds)) return seconds * 1000;
  const date = Date.parse(header);
  if (!Number.isNaN(date)) return Math.max(0, date - Date.now());
  return null;
}

Provider-Specific Headers

Beyond Retry-After, providers expose remaining-quota and reset-time headers:

• OpenAI: x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, x-ratelimit-reset-requests, x-ratelimit-reset-tokens (the latter as ISO 8601 durations like 60s).
• GitHub: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (Unix timestamp).
• Stripe: Retry-After only on rate-limit responses.
• Twilio: Retry-After on overload; concurrent request limits surfaced as 429 with explanatory body.
• SendGrid: X-RateLimit-Remaining, X-RateLimit-Reset.

Always check the provider's docs and use what they give you. Falling back to your own backoff math is the safety net, not the primary plan.

Exponential Backoff With Full Jitter

When you don't have a Retry-After to honor, the backoff math you want is exponential backoff with full jitter. The canonical reference is the AWS Architecture Blog post Exponential Backoff and Jitter by Marc Brooker.

The naive version (don't use):

// BAD — no jitter, synchronized retries
const delay = Math.pow(2, attempt) * baseMs;

The full-jitter version (use this):

// GOOD — full jitter
const delay = Math.random() * Math.min(maxMs, Math.pow(2, attempt) * baseMs);

With baseMs = 100 and maxMs = 30000, retries fall in: [0,200], [0,400], [0,800], [0,1600], [0,3200], etc. Distributing across the entire window (rather than [X/2, X]) decorrelates retries across all clients and is the AWS-blessed default.

A complete TypeScript retry helper

type RetryOptions = {
  maxAttempts?: number;
  baseMs?: number;
  maxMs?: number;
  isRetryable?: (err: unknown) => boolean;
};

async function withRetry<T>(
  fn: () => Promise<T>,
  opts: RetryOptions = {}
): Promise<T> {
  const {
    maxAttempts = 4,
    baseMs = 200,
    maxMs = 30_000,
    isRetryable = defaultIsRetryable,
  } = opts;

  let lastErr: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      if (!isRetryable(err) || attempt === maxAttempts - 1) throw err;

      const retryAfter = extractRetryAfter(err);
      const backoff = Math.random() * Math.min(maxMs, baseMs * 2 ** attempt);
      const delay = retryAfter ?? backoff;

      await new Promise((r) => setTimeout(r, delay));
    }
  }
  throw lastErr;
}

function defaultIsRetryable(err: unknown): boolean {
  if (!(err instanceof Error)) return false;
  const status = (err as any).status ?? (err as any).response?.status;
  if (status === 429) return true;
  if (status >= 500 && status < 600) return true;
  if ((err as any).code === "ECONNRESET") return true;
  if ((err as any).code === "ETIMEDOUT") return true;
  return false;
}

Python equivalent

import asyncio
import random
from typing import Awaitable, Callable, TypeVar

T = TypeVar("T")

async def with_retry(
    fn: Callable[[], Awaitable[T]],
    max_attempts: int = 4,
    base_ms: int = 200,
    max_ms: int = 30_000,
) -> T:
    last_err: Exception | None = None
    for attempt in range(max_attempts):
        try:
            return await fn()
        except Exception as err:
            last_err = err
            if not is_retryable(err) or attempt == max_attempts - 1:
                raise
            retry_after = extract_retry_after(err)
            backoff = random.random() * min(max_ms, base_ms * (2 ** attempt))
            await asyncio.sleep((retry_after or backoff) / 1000)
    assert last_err is not None
    raise last_err

Circuit Breakers: Failing Fast When the Upstream Is Down

Retry logic helps when failures are transient. When an upstream is fully down, retries make things worse. Each retry occupies a worker for 30 seconds (the timeout). Your worker pool fills with stuck retries. Your service can't even respond to unrelated user requests.

A circuit breaker fixes this. It monitors error rate over a window and "opens" — failing all requests immediately — when the rate exceeds a threshold. It periodically tries again ("half-open"), and reverts to "closed" when the upstream recovers.

Node — using opossum

import CircuitBreaker from "opossum";

const breaker = new CircuitBreaker(callStripe, {
  timeout: 5000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
  rollingCountTimeout: 10000,
  rollingCountBuckets: 10,
});

breaker.on("open", () => log.warn("Stripe circuit OPEN"));
breaker.on("halfOpen", () => log.info("Stripe circuit HALF-OPEN"));
breaker.on("close", () => log.info("Stripe circuit CLOSED"));

// Use it
const result = await breaker.fire(payload);

Python — using pybreaker

import pybreaker

stripe_breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=30,
    exclude=[ValidationError],
)

@stripe_breaker
def call_stripe(payload):
    return stripe.PaymentIntent.create(**payload)

The Hystrix model — popularized by Netflix and documented at Microsoft's Cloud Design Patterns — is the conceptual reference if you want to dig deeper.

Queue-Based Architectures: Concurrency Control at the Right Layer

For any non-trivial volume of API calls, the right architecture is a queue with rate-limited workers. Your application enqueues jobs; workers consume them at a controlled rate.

BullMQ for Node

import { Queue, Worker } from "bullmq";

const queue = new Queue("openai-jobs", { connection: redis });

const worker = new Worker(
  "openai-jobs",
  async (job) => {
    return await withRetry(() => callOpenAI(job.data), {
      isRetryable: (e) => isOpenAIRetryable(e),
    });
  },
  {
    connection: redis,
    concurrency: 4,                    // 4 in-flight requests
    limiter: { max: 50, duration: 60_000 }, // global 50 req/min cap
  }
);

The concurrency and limiter settings give you per-worker concurrency and global rate. You can run multiple worker processes; the limiter is enforced across all of them via Redis.

Python — Celery / Dramatiq / arq

For Python, Celery's rate_limit per task and Dramatiq's middleware-based throttle work similarly. arq has max_jobs per worker. The pattern is the same: control the rate where workers consume, not where producers enqueue.

SQS + Lambda

For serverless, AWS SQS plus Lambda's ReservedConcurrentExecutions is the canonical setup. Set the reserved concurrency to whatever rate the upstream tolerates. SQS's visibility timeout serves as the implicit retry: if the worker fails to delete the message within the window, SQS redelivers.

For the AI-specific case (OpenAI, Anthropic), the recommendations in the OpenAI cookbook on rate limit handling are the gold standard.

Idempotency: Mandatory for Retried POST

Retrying a GET is safe. Retrying a POST is dangerous unless the operation is idempotent.

Generate an idempotency key per logical operation:

import { randomUUID } from "crypto";

const idempotencyKey = randomUUID();

await withRetry(() =>
  stripe.paymentIntents.create(payload, { idempotencyKey })
);

Stripe stores the idempotency key for 24 hours. The first request executes; subsequent requests with the same key return the original response. This is the difference between a clean retry and a duplicate charge.

OpenAI exposes an Idempotency-Key header. Twilio exposes Idempotency-Token on most write endpoints. SendGrid uses message IDs. Always use them on POSTs that retry.

For more on the agent-specific case where AI workflows generate the API calls, see why your AI agent loses context and how to fix it — context loss across retries is a particularly nasty interaction.

Provider-Specific Quirks

OpenAI

• Two limit dimensions: RPM (requests per minute) and TPM (tokens per minute).
• Headers: x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, x-ratelimit-reset-requests, x-ratelimit-reset-tokens.
• Reset format: ISO 8601 duration string (60s, 1m, 6h).
• Tier-based limits. Higher usage history unlocks higher limits.
• Streaming responses count differently. Tokens are billed as generated, not as estimated.

For batch workloads, OpenAI's Batch API gives 50% discounts at the cost of up to 24h latency — the right tool when latency isn't critical.

Stripe

• 100 read + 100 write requests per second per account.
• Burst tolerance is high — Stripe absorbs short spikes.
• Idempotency keys mandatory for safe retry of payment creation.
• Test mode and live mode have separate limits.

Twilio

• Concurrency-based. Total in-flight requests per account.
• Per-phone-number throughput. Long codes ~1 SMS/sec; toll-free and short codes higher.
• A2P 10DLC has its own throughput rules in the US.

SendGrid

• IP warmup-based. New IPs throttle to a few hundred sends/day; ramps up over weeks.
• Account category limits. Marketing vs transactional separate.
• Burst rejections return 429 with X-RateLimit-Reset.

Always read each provider's API documentation. Generic retry libraries fail when they assume one model.

Monitoring: What to Alert On

Bad alerting causes more pages than bad code. The metrics that matter for retry behavior:

1. Retry rate per upstream — sudden increases indicate upstream degradation. Alert when > 5x baseline for 5 minutes.
2. Success-after-retry rate — high (>90%) means retries are working. Low (<50%) means retries aren't helping and you have a deeper issue.
3. Permanent failures — requests that exhausted all retries. These are user-visible. Alert on absolute rate.
4. Circuit breaker state — every state change should be logged. Page on "open" lasting > 5 minutes.
5. Queue depth and age — for queue-based architectures, queue depth growing without recovery is the canonical "we're behind" signal.

In Sentry, attach retry context as tags:

Sentry.setTag("retry.attempt", String(attempt));
Sentry.setTag("retry.upstream", "openai");
Sentry.setTag("retry.reason", String(status));

In Datadog, custom metrics with the same dimensions plus dashboards for the four metrics above.

What Good Looks Like

A production-grade external API call in 2026 has this shape:

1. Idempotency key generated once per logical operation.
2. Wrapped in a circuit breaker to fail fast when upstream is down.
3. Retries handled by an inner helper with full-jitter exponential backoff and proper Retry-After parsing.
4. Submitted via a rate-limited queue with concurrency aligned to the upstream's tolerance.
5. Observability tags for retry attempt, reason, and upstream.
6. Alerts on retry rate, permanent failures, and circuit state changes — not raw error counts.

That sounds like a lot. In practice, it is 200 lines of code that you write once, abstract behind a client wrapper, and reuse for every external dependency. The night you don't get paged at 3 AM is the night you realize it was worth it.

For more on developer tooling that makes building this kind of resilience faster, see our best AI tools for developers in 2026 — modern AI coding assistants reduce the friction of writing and reviewing the kind of error-handling code that, historically, no one wanted to write.

A Final Sanity Check

Before you ship retry logic, ask:

• Does it parse Retry-After correctly for both seconds and date forms?
• Does it use full jitter, not equal jitter?
• Does it have a maximum attempt count?
• Does it have a maximum backoff?
• Does it differentiate retryable from non-retryable errors?
• Does it carry an idempotency key on POSTs?
• Is it wrapped in a circuit breaker?
• Are retries metered by tag in your observability?

If any of those answers is "no," you have homework. If all of them are "yes," you have something that survives 3 AM.

For the wider context on shipping production-grade backend systems, see our pillar guide: [Best AI Tools for Developers in 2026](/blog/best-ai-tools-for-developers-2026).