Adamant Tools

Adamant Tools › Blog

  • APIs
  • Python
  • JavaScript
  • Reliability
  • Streaming

Timeouts and cancellation for LLM calls: idle timeout, total deadline, stop button (Python, Node.js)

A model call can be slow, silent or endless. Four kinds of timeout, why a read timeout never stops a streaming answer, and tested Python and Node.js code with a cancel.

Published 7 min readGitHubLeggi in italiano

Timeouts and cancellation for LLM calls: idle timeout, total deadline, stop button (Python, Node.js)
Foto: Jacek Rużyczka, CC BY-SA 4.0, Wikimedia Commons

A normal HTTP call either answers in milliseconds or fails. A call to a language model is different: the first token can take many seconds, the answer can keep arriving for minutes, and a stuck upstream can stay silent while your connection stays open. If your code has no clear time limits, one slow request can hold a thread, a connection and a user's patience indefinitely.

This guide separates the timeouts you need, shows why the usual "timeout" setting does not protect a streaming answer, and gives tested code for Python (httpx) and Node.js (fetch). Both clients support three ways to end a call: an idle timeout, a total deadline, and cancellation by the caller (a "Stop" button). The tests run against a local fake server, so they need no API key. If you also retry failed calls, read HTTP 429 from AI APIs: retry with backoff and Retry-After too: timeouts and retries have to be designed together.

Versions checked on 2026-11-10: httpx 0.28.1 on PyPI, Python 3.11 or newer, Node.js 20.3 or newer (the tests ran on 22). Java is not covered in this article.

Four different timeouts

"Timeout" means different things at different layers. For one model call you want to think about four:

Timeout What it limits Typical failure it catches
Connect Opening the TCP/TLS connection Wrong host, firewall dropping packets
Time to first byte Waiting for the first part of the answer The model is overloaded or still "thinking"
Idle (read) Silence between two chunks of data An upstream that froze mid-answer
Total deadline The whole call, from start to last byte A stream that never ends, or a very slow one

Most HTTP libraries give you the first three in some form and leave the fourth to you. The Python library httpx documents separate connect, read, write and pool timeouts in its timeouts guide. In the browser and in Node.js, fetch has no timeout option at all: you pass an AbortSignal.

Why a read timeout does not stop a streaming answer

A read timeout is the longest silence the client accepts. It is reset every time data arrives. For a streamed answer (see how to stream one to the browser) the server sends a token every few milliseconds, so the timer never expires, even if the answer is endless or the model is generating a huge text nobody wants any more.

In other words, the read timeout answers "has the upstream gone quiet?", not "has this call taken too long?". You need both questions, and the second one needs a separate total deadline that does not reset.

A second pitfall: libraries have their own defaults. The httpx documentation states that by default it raises an exception after 5 seconds of network inactivity. That is fine for a normal API, but a model that thinks for ten seconds before its first token will fail on every request unless you raise the read timeout for that client. The opposite default, no timeout at all, exists in other libraries and is worse: the call can hang forever.

Pick the numbers from measurements, not from guesses. Log time to first token and total duration for a few days, then set the idle timeout above the slowest normal first token and the deadline above the slowest normal answer.

Python: httpx plus asyncio.timeout

python
import asyncio

import httpx

# Four separate limits. "read" is the longest silence allowed between two chunks,
# not the time for the whole answer.
TIMEOUTS = httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0)


async def stream_completion(url, payload, total_deadline=120.0, on_token=print):
    """Stream lines from a model endpoint with a per-chunk timeout AND a total deadline.

    Leaving the `async with` blocks (deadline, error or task.cancel()) closes the
    connection, which is the signal the server needs to stop generating.
    """
    tokens = []
    async with httpx.AsyncClient(timeout=TIMEOUTS) as client:
        async with asyncio.timeout(total_deadline):  # Python 3.11+
            async with client.stream("POST", url, json=payload) as resp:
                resp.raise_for_status()
                async for line in resp.aiter_lines():
                    tokens.append(line)
                    on_token(line)
    return tokens

What the code does:

  • TIMEOUTS sets four separate limits. Here read=30.0 is the longest silence between two chunks of the response body, not the length of the answer.
  • asyncio.timeout(total_deadline) is the total deadline. It was added in Python 3.11 (see the asyncio documentation) and raises the built-in TimeoutError when the time is up.
  • client.stream(...) does not read the body into memory; it hands you the lines as they arrive.
  • When the code leaves the async with blocks for any reason (deadline, error, or task.cancel()), httpx closes the connection. That closing is the only "stop" message the server gets.

Note the two different exception types. A read timeout raises httpx.ReadTimeout, a subclass of httpx.TimeoutException; the deadline raises the built-in TimeoutError. Catch both if you want to treat them the same way, and do not catch the generic Exception: asyncio.CancelledError is not a TimeoutError, and swallowing it would break cancellation.

JavaScript: AbortSignal.timeout and AbortSignal.any

javascript
// Node.js 20.3+ (AbortSignal.any). No dependencies.
export async function streamCompletion(url, payload, { totalMs = 120_000, idleMs = 30_000, signal, onToken = console.log } = {}) {
  // One controller we abort ourselves when the stream goes quiet.
  const idle = new AbortController();
  let idleTimer;
  const armIdle = () => {
    clearTimeout(idleTimer);
    idleTimer = setTimeout(() => idle.abort(new Error(`no data for ${idleMs} ms`)), idleMs);
  };

  const signals = [AbortSignal.timeout(totalMs), idle.signal];  // total deadline + idle timeout
  if (signal) signals.push(signal);                              // + cancellation from the caller (a "Stop" button)

  const tokens = [];
  armIdle();
  try {
    const res = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(payload),
      signal: AbortSignal.any(signals),
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const decoder = new TextDecoder();
    let buffer = '';
    for await (const chunk of res.body) {
      armIdle();                                                 // data arrived: restart the idle clock
      buffer += decoder.decode(chunk, { stream: true });
      const lines = buffer.split('\n');
      buffer = lines.pop();
      for (const line of lines) { tokens.push(line); onToken(line); }
    }
    return tokens;
  } finally {
    clearTimeout(idleTimer);
  }
}

Here the three limits are three signals combined into one:

  • AbortSignal.timeout(ms) is the total deadline. It aborts with a TimeoutError.
  • idle is an AbortController that we abort ourselves from a timer. The timer is re-armed every time a chunk arrives, which makes it an idle timeout.
  • The signal passed by the caller is the "Stop" button: controller.abort() ends the request and the loop with an AbortError.
  • AbortSignal.any([...]) merges them. It needs Node.js 20.3 or newer (or 18.17), as the Node.js documentation states; in browsers check the compatibility table on MDN before using it.

One detail that is easy to forget: the idle timer must also run before the first chunk (we call armIdle() before fetch), otherwise a server that accepts the connection and never answers is only stopped by the total deadline. And the finally clears the timer, or a finished call would keep the Node.js process alive until the timer fires.

Cancellation really means closing the connection

HTTP has no "cancel" message. When you abort a request, the client closes the connection (or resets the stream with HTTP/2). Whether the model stops generating depends on the server noticing that. Many servers notice only when they try to write the next chunk and get an error; some proxies keep reading from the upstream anyway. If you are billed per generated token, an abandoned call may still be partly billed. Check how your provider and your own gateway behave, and do not assume that a cancel is free.

The fake server in the example shows the mechanism in a few lines: it writes a token every 0.1 seconds and counts a stream as abandoned when the write fails.

python
    def _stream(self, stall):
        self.send_response(200)
        self.send_header("content-type", "text/plain")
        self.end_headers()
        try:
            n = 0
            while True:
                n += 1
                self.wfile.write(f"token {n}\n".encode())
                self.wfile.flush()
                if stall:
                    time.sleep(30)
                    return
                time.sleep(0.1)
        except (BrokenPipeError, ConnectionResetError):
            with lock:
                stats["abandoned"] += 1

The tests

Three tests per language, with the same three scenarios. /drip sends a token every 0.1 s forever, /stall sends one line and then goes silent for 30 s. This is the Python test for the total deadline:

python
    async def test_total_deadline_stops_a_stream_that_never_ends(self):
        before = abandoned(self.base)
        got = []
        # Tokens keep arriving every 0.1 s, so the read timeout (30 s) never fires.
        with self.assertRaises(TimeoutError):
            await client.stream_completion(f"{self.base}/drip", {}, total_deadline=1.0, on_token=got.append)
        self.assertGreater(len(got), 3)  # it did receive tokens before the deadline
        self.assertTrue(await wait_for_abandoned(self.base, before + 1), "server never saw the disconnect")

The test proves two things: the deadline fires although data keeps arriving, and the server really saw the connection close. The other tests check that the read timeout (httpx.ReadTimeout, or the idle timeout in JavaScript) ends the stalled call, and that cancelling the task or aborting the signal also closes the connection. The Node.js tests also run the same Python fake server, so both clients are checked against identical behaviour.

Things I noticed while writing the tests:

  • The disconnect is detected by the server with a short delay, so the test polls /stats for a few seconds instead of checking once. A test that checks immediately is flaky.
  • The connect timeout is configured in the code but not tested: reproducing it needs an address that silently drops packets, which is not reliable on a CI runner.
  • /stall makes the server hold a thread for 30 s. The fake server uses daemon threads so the test run does not wait for them.

You can run everything with python3 -m unittest -v and node --test in the example folder. The CI run for this article is green.

Timeouts and retries

When a timeout fires, the next question is whether to retry. A few rules that hold in practice:

  • Retry connect timeouts: nothing was sent. Retry idle timeouts only if the request is safe to repeat, and remember that the model may have processed the first one. See the 429 article on how a retried call is a new, billable call.
  • A total-deadline failure usually means the work is too big for the limit. Retrying the same request with the same limit will fail the same way; shorten the prompt or the maximum output instead.
  • Add waiting time between attempts with exponential backoff and jitter; the retry and backoff simulator lets you see what a given policy does.
  • Make the budget visible: if a user-facing request has a 30 s limit, the sum of all attempts and waits must fit in it. Pass the remaining budget down instead of giving every attempt a fresh 30 s.

If the failure arrives as an HTTP status rather than a timeout, our HTTP status code reference explains which ones are worth retrying.

Summary

  • A model call needs four limits: connect, first byte, idle and total.
  • A read or idle timeout never ends a stream that keeps sending data; add a total deadline.
  • In httpx combine httpx.Timeout(...) with asyncio.timeout(...); in Node.js combine AbortSignal.timeout, a re-armed idle timer and the caller's signal with AbortSignal.any.
  • Cancelling means closing the connection. Whether the server stops generating, and whether you are still billed, depends on the server and on any proxy in between: test it with your own provider.
  • Measure real latencies before choosing numbers, and keep the total budget of timeouts plus retries inside what the user will accept.

Sources: httpx timeouts, Python asyncio.timeout, MDN AbortSignal, Node.js globals, RFC 9110, HTTP Semantics.

Full code on GitHub
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/timeouts

About the author

Written by the Adamant Solutions engineering team: senior engineers working on APIs, identity and enterprise integration, certified on WSO2 and Spring. Code is tested before publication; if you spot an error, write to administrata@adamant-solutions.com.