Adamant Tools

Adamant Tools › Blog

  • APIs
  • Python
  • JavaScript
  • Java
  • Reliability

HTTP 429 from AI APIs: retry with backoff and Retry-After (Python, Node.js, Java)

When a model API answers 429 Too Many Requests, retrying immediately makes things worse. What the standards say, which errors to retry, and tested retry code with no dependencies in Python, JavaScript and Java.

Published 3 min readGitHubLeggi in italiano

HTTP 429 from AI APIs: retry with backoff and Retry-After (Python, Node.js, Java)
Foto: Jacek Rużyczka, CC BY-SA 4.0, Wikimedia Commons

Sooner or later every application that calls a language-model API gets this answer:

http
HTTP/1.1 429 Too Many Requests
Retry-After: 20

AI APIs limit how many requests and how many tokens you can send per minute, and a single busy feature can hit those limits. The naive fix, retrying at once in a loop, sends even more requests into a service that has just asked you to slow down. This guide shows a retry strategy that works, with code you can paste into Python, Node.js or Java. All three versions use only the standard library and pass the same test.

What 429 and Retry-After mean

Two standards define the behaviour:

  • RFC 6585, section 4 defines 429 Too Many Requests: the client sent too many requests in a given amount of time. The response may include a Retry-After header.
  • RFC 9110, section 10.2.3 defines Retry-After. Its value is either a number of seconds (Retry-After: 20) or an HTTP date (Retry-After: Fri, 31 Oct 2026 09:00:00 GMT).

Your code must handle both forms. Many clients only parse the number and silently ignore the date form.

Which errors to retry

Status Retry? Why
429 Too Many Requests Yes, after waiting You hit a rate limit; it will reset.
500, 502, 503, 504 Yes, a few times Temporary server or gateway problems.
400, 422 No The request is wrong (bad JSON, prompt too long). Retrying sends the same mistake.
401, 403 No Missing or invalid key, or no permission. Fix the configuration.
404 No Wrong URL or model name.

Network errors and timeouts can also be retried, with the same backoff. Our HTTP status codes reference explains each code in more detail.

How long to wait

  1. If the server sends Retry-After, use it. The server knows when your limit resets; you don't.
  2. Otherwise use exponential backoff with jitter. The ceiling doubles at each attempt (0.5 s, 1 s, 2 s, 4 s…), and the actual wait is a random value between zero and that ceiling ("full jitter"). The randomness stops many clients that failed together from retrying together. AWS describes and compares these strategies in Exponential Backoff And Jitter.
  3. Cap the wait and the number of attempts. A user is waiting on the other side; after a few attempts, fail with a clear error.

You can try different settings in our retry and backoff simulator, which shows when each retry would fire and how load spreads over time.

Python (standard library)

python
import email.utils
import json
import random
import time
import urllib.error
import urllib.request

RETRYABLE = {429, 500, 502, 503, 504}


def retry_after_seconds(value):
    """Retry-After is either a number of seconds or an HTTP date (RFC 9110, section 10.2.3)."""
    if not value:
        return None
    value = value.strip()
    if value.isdigit():
        return int(value)
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, when.timestamp() - time.time())


def post_json(url, payload, headers=None, max_attempts=5, base=0.5, cap=30.0):
    body = json.dumps(payload).encode()
    for attempt in range(1, max_attempts + 1):
        req = urllib.request.Request(url, data=body, method="POST",
                                     headers={"content-type": "application/json", **(headers or {})})
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                return json.loads(resp.read())
        except urllib.error.HTTPError as err:
            if err.code not in RETRYABLE or attempt == max_attempts:
                raise
            wait = retry_after_seconds(err.headers.get("Retry-After"))
            if wait is None:
                # Exponential backoff with full jitter: random value between 0 and base * 2^attempt, capped.
                wait = random.uniform(0, min(cap, base * 2 ** attempt))
            wait = min(wait, cap)
            print(f"HTTP {err.code}, attempt {attempt}/{max_attempts}, waiting {wait:.1f}s")
            time.sleep(wait)

If you use httpx or requests, the logic is the same: read the status code and the Retry-After header from the response and wait before the next attempt.

JavaScript (Node.js 18+)

javascript
const RETRYABLE = new Set([429, 500, 502, 503, 504]);

export function retryAfterMs(value) {
  if (!value) return null;
  const v = value.trim();
  if (/^\d+$/.test(v)) return Number(v) * 1000;          // seconds
  const date = Date.parse(v);                             // or an HTTP date
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function postJson(url, payload, { headers = {}, maxAttempts = 5, baseMs = 500, capMs = 30_000 } = {}) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const res = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json', ...headers },
      body: JSON.stringify(payload),
      signal: AbortSignal.timeout(60_000),
    });
    if (res.ok) return res.json();
    if (!RETRYABLE.has(res.status) || attempt === maxAttempts) {
      throw new Error(`HTTP ${res.status}: ${await res.text()}`);
    }
    let wait = retryAfterMs(res.headers.get('retry-after'));
    if (wait === null) wait = Math.random() * Math.min(capMs, baseMs * 2 ** attempt); // full jitter
    wait = Math.min(wait, capMs);
    console.log(`HTTP ${res.status}, attempt ${attempt}/${maxAttempts}, waiting ${(wait / 1000).toFixed(1)}s`);
    await res.body?.cancel();
    await sleep(wait);
  }
}

res.body?.cancel() releases the unread error body before the next attempt, so connections are not kept busy.

Java (11+, java.net.http)

java
private static final Set<Integer> RETRYABLE = Set.of(429, 500, 502, 503, 504);
private final HttpClient http = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();

static Optional<Duration> retryAfter(String value) {
    if (value == null || value.isBlank()) return Optional.empty();
    String v = value.trim();
    if (v.chars().allMatch(Character::isDigit)) return Optional.of(Duration.ofSeconds(Long.parseLong(v)));
    try {
        Instant when = ZonedDateTime.parse(v, DateTimeFormatter.RFC_1123_DATE_TIME).toInstant();
        Duration d = Duration.between(Instant.now(), when);
        return Optional.of(d.isNegative() ? Duration.ZERO : d);
    } catch (DateTimeParseException e) {
        return Optional.empty();
    }
}

public String postJson(String url, String json, int maxAttempts) throws IOException, InterruptedException {
    long baseMs = 500, capMs = 30_000;
    for (int attempt = 1; ; attempt++) {
        HttpRequest req = HttpRequest.newBuilder(URI.create(url))
                .timeout(Duration.ofSeconds(60))
                .header("content-type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(json))
                .build();
        HttpResponse<String> res = http.send(req, HttpResponse.BodyHandlers.ofString());
        if (res.statusCode() < 300) return res.body();
        if (!RETRYABLE.contains(res.statusCode()) || attempt == maxAttempts) {
            throw new IOException("HTTP " + res.statusCode() + ": " + res.body());
        }
        long ceiling = Math.min(capMs, baseMs << attempt);
        long waitMs = retryAfter(res.headers().firstValue("retry-after").orElse(null))
                .map(Duration::toMillis)
                .orElseGet(() -> ThreadLocalRandom.current().nextLong(ceiling + 1)); // full jitter
        waitMs = Math.min(waitMs, capMs);
        Thread.sleep(waitMs);
    }
}

DateTimeFormatter.RFC_1123_DATE_TIME parses the HTTP date format used by Retry-After. If you use Spring AI instead of calling the API yourself, retries are built in; check its defaults in our Spring AI guide.

Test it without spending a cent

Don't wait for a real rate limit to find out whether your retry code works. The repository includes a 30-line test server that answers 429 with Retry-After: 1, then 429 with an HTTP date, then 503 without the header, then 200. All three clients above pass against it:

text
HTTP 429, attempt 1/5, waiting 1.0s
HTTP 429, attempt 2/5, waiting 0.6s
HTTP 503, attempt 3/5, waiting 0.3s
{'ok': True, 'attempts': 4}

Run ./test_retry.sh in the example folder to repeat the test, or point the HTTP request tester at your own endpoint to see its real headers.

Retries are not a rate limiter

Retries recover from occasional limits. If you hit 429 all the time, slow down on your side:

  • limit concurrent requests (a small pool or a semaphore in front of the API client);
  • queue background work instead of sending it all at once;
  • send fewer tokens: shorter prompts, smaller context, cached answers for repeated questions;
  • read the rate-limit headers your provider returns (many send the remaining requests and tokens) and slow down before you reach zero.

One more caution: a retried request is a new request. With AI APIs, a call that timed out on your side may still have been processed and billed. Keep the number of attempts small, and log every retry so you can see it in your costs.

Full code on GitHub
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/retry-429

About the author

Written by the Adamant Solutions engineering team: senior engineers working on APIs, identity and enterprise integration, certified on WSO2 and Spring. Code is tested before publication; if you spot an error, write to administrata@adamant-solutions.com.