Timeouts and cancellation for LLM calls: idle timeout, total deadline, stop button (Python, Node.js)
A model call can be slow, silent or endless. Four kinds of timeout, why a read timeout never stops a streaming answer, and tested Python and Node.js code with a cancel.

A normal HTTP call either answers in milliseconds or fails. A call to a language model is different: the first token can take many seconds, the answer can keep arriving for minutes, and a stuck upstream can stay silent while your connection stays open. If your code has no clear time limits, one slow request can hold a thread, a connection and a user's patience indefinitely.
This guide separates the timeouts you need, shows why the usual "timeout" setting does not protect a streaming answer, and gives tested code for Python (httpx) and Node.js (fetch). Both clients support three ways to end a call: an idle timeout, a total deadline, and cancellation by the caller (a "Stop" button). The tests run against a local fake server, so they need no API key. If you also retry failed calls, read HTTP 429 from AI APIs: retry with backoff and Retry-After too: timeouts and retries have to be designed together.
Versions checked on 2026-11-10: httpx 0.28.1 on PyPI, Python 3.11 or newer, Node.js 20.3 or newer (the tests ran on 22). Java is not covered in this article.
Four different timeouts
"Timeout" means different things at different layers. For one model call you want to think about four:
| Timeout | What it limits | Typical failure it catches |
|---|---|---|
| Connect | Opening the TCP/TLS connection | Wrong host, firewall dropping packets |
| Time to first byte | Waiting for the first part of the answer | The model is overloaded or still "thinking" |
| Idle (read) | Silence between two chunks of data | An upstream that froze mid-answer |
| Total deadline | The whole call, from start to last byte | A stream that never ends, or a very slow one |
Most HTTP libraries give you the first three in some form and leave the fourth to you. The Python library httpx documents separate connect, read, write and pool timeouts in its timeouts guide. In the browser and in Node.js, fetch has no timeout option at all: you pass an AbortSignal.
Why a read timeout does not stop a streaming answer
A read timeout is the longest silence the client accepts. It is reset every time data arrives. For a streamed answer (see how to stream one to the browser) the server sends a token every few milliseconds, so the timer never expires, even if the answer is endless or the model is generating a huge text nobody wants any more.
In other words, the read timeout answers "has the upstream gone quiet?", not "has this call taken too long?". You need both questions, and the second one needs a separate total deadline that does not reset.
A second pitfall: libraries have their own defaults. The httpx documentation states that by default it raises an exception after 5 seconds of network inactivity. That is fine for a normal API, but a model that thinks for ten seconds before its first token will fail on every request unless you raise the read timeout for that client. The opposite default, no timeout at all, exists in other libraries and is worse: the call can hang forever.
Pick the numbers from measurements, not from guesses. Log time to first token and total duration for a few days, then set the idle timeout above the slowest normal first token and the deadline above the slowest normal answer.
Python: httpx plus asyncio.timeout
import asyncio
import httpx
# Four separate limits. "read" is the longest silence allowed between two chunks,
# not the time for the whole answer.
TIMEOUTS = httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0)
async def stream_completion(url, payload, total_deadline=120.0, on_token=print):
"""Stream lines from a model endpoint with a per-chunk timeout AND a total deadline.
Leaving the `async with` blocks (deadline, error or task.cancel()) closes the
connection, which is the signal the server needs to stop generating.
"""
tokens = []
async with httpx.AsyncClient(timeout=TIMEOUTS) as client:
async with asyncio.timeout(total_deadline): # Python 3.11+
async with client.stream("POST", url, json=payload) as resp:
resp.raise_for_status()
async for line in resp.aiter_lines():
tokens.append(line)
on_token(line)
return tokens
What the code does:
TIMEOUTSsets four separate limits. Hereread=30.0is the longest silence between two chunks of the response body, not the length of the answer.asyncio.timeout(total_deadline)is the total deadline. It was added in Python 3.11 (see the asyncio documentation) and raises the built-inTimeoutErrorwhen the time is up.client.stream(...)does not read the body into memory; it hands you the lines as they arrive.- When the code leaves the
async withblocks for any reason (deadline, error, ortask.cancel()), httpx closes the connection. That closing is the only "stop" message the server gets.
Note the two different exception types. A read timeout raises httpx.ReadTimeout, a subclass of httpx.TimeoutException; the deadline raises the built-in TimeoutError. Catch both if you want to treat them the same way, and do not catch the generic Exception: asyncio.CancelledError is not a TimeoutError, and swallowing it would break cancellation.
JavaScript: AbortSignal.timeout and AbortSignal.any
// Node.js 20.3+ (AbortSignal.any). No dependencies.
export async function streamCompletion(url, payload, { totalMs = 120_000, idleMs = 30_000, signal, onToken = console.log } = {}) {
// One controller we abort ourselves when the stream goes quiet.
const idle = new AbortController();
let idleTimer;
const armIdle = () => {
clearTimeout(idleTimer);
idleTimer = setTimeout(() => idle.abort(new Error(`no data for ${idleMs} ms`)), idleMs);
};
const signals = [AbortSignal.timeout(totalMs), idle.signal]; // total deadline + idle timeout
if (signal) signals.push(signal); // + cancellation from the caller (a "Stop" button)
const tokens = [];
armIdle();
try {
const res = await fetch(url, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(payload),
signal: AbortSignal.any(signals),
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const decoder = new TextDecoder();
let buffer = '';
for await (const chunk of res.body) {
armIdle(); // data arrived: restart the idle clock
buffer += decoder.decode(chunk, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop();
for (const line of lines) { tokens.push(line); onToken(line); }
}
return tokens;
} finally {
clearTimeout(idleTimer);
}
}
Here the three limits are three signals combined into one:
AbortSignal.timeout(ms)is the total deadline. It aborts with aTimeoutError.idleis anAbortControllerthat we abort ourselves from a timer. The timer is re-armed every time a chunk arrives, which makes it an idle timeout.- The
signalpassed by the caller is the "Stop" button:controller.abort()ends the request and the loop with anAbortError. AbortSignal.any([...])merges them. It needs Node.js 20.3 or newer (or 18.17), as the Node.js documentation states; in browsers check the compatibility table on MDN before using it.
One detail that is easy to forget: the idle timer must also run before the first chunk (we call armIdle() before fetch), otherwise a server that accepts the connection and never answers is only stopped by the total deadline. And the finally clears the timer, or a finished call would keep the Node.js process alive until the timer fires.
Cancellation really means closing the connection
HTTP has no "cancel" message. When you abort a request, the client closes the connection (or resets the stream with HTTP/2). Whether the model stops generating depends on the server noticing that. Many servers notice only when they try to write the next chunk and get an error; some proxies keep reading from the upstream anyway. If you are billed per generated token, an abandoned call may still be partly billed. Check how your provider and your own gateway behave, and do not assume that a cancel is free.
The fake server in the example shows the mechanism in a few lines: it writes a token every 0.1 seconds and counts a stream as abandoned when the write fails.
def _stream(self, stall):
self.send_response(200)
self.send_header("content-type", "text/plain")
self.end_headers()
try:
n = 0
while True:
n += 1
self.wfile.write(f"token {n}\n".encode())
self.wfile.flush()
if stall:
time.sleep(30)
return
time.sleep(0.1)
except (BrokenPipeError, ConnectionResetError):
with lock:
stats["abandoned"] += 1
The tests
Three tests per language, with the same three scenarios. /drip sends a token every 0.1 s forever, /stall sends one line and then goes silent for 30 s. This is the Python test for the total deadline:
async def test_total_deadline_stops_a_stream_that_never_ends(self):
before = abandoned(self.base)
got = []
# Tokens keep arriving every 0.1 s, so the read timeout (30 s) never fires.
with self.assertRaises(TimeoutError):
await client.stream_completion(f"{self.base}/drip", {}, total_deadline=1.0, on_token=got.append)
self.assertGreater(len(got), 3) # it did receive tokens before the deadline
self.assertTrue(await wait_for_abandoned(self.base, before + 1), "server never saw the disconnect")
The test proves two things: the deadline fires although data keeps arriving, and the server really saw the connection close. The other tests check that the read timeout (httpx.ReadTimeout, or the idle timeout in JavaScript) ends the stalled call, and that cancelling the task or aborting the signal also closes the connection. The Node.js tests also run the same Python fake server, so both clients are checked against identical behaviour.
Things I noticed while writing the tests:
- The disconnect is detected by the server with a short delay, so the test polls
/statsfor a few seconds instead of checking once. A test that checks immediately is flaky. - The connect timeout is configured in the code but not tested: reproducing it needs an address that silently drops packets, which is not reliable on a CI runner.
/stallmakes the server hold a thread for 30 s. The fake server uses daemon threads so the test run does not wait for them.
You can run everything with python3 -m unittest -v and node --test in the example folder. The CI run for this article is green.
Timeouts and retries
When a timeout fires, the next question is whether to retry. A few rules that hold in practice:
- Retry connect timeouts: nothing was sent. Retry idle timeouts only if the request is safe to repeat, and remember that the model may have processed the first one. See the 429 article on how a retried call is a new, billable call.
- A total-deadline failure usually means the work is too big for the limit. Retrying the same request with the same limit will fail the same way; shorten the prompt or the maximum output instead.
- Add waiting time between attempts with exponential backoff and jitter; the retry and backoff simulator lets you see what a given policy does.
- Make the budget visible: if a user-facing request has a 30 s limit, the sum of all attempts and waits must fit in it. Pass the remaining budget down instead of giving every attempt a fresh 30 s.
If the failure arrives as an HTTP status rather than a timeout, our HTTP status code reference explains which ones are worth retrying.
Summary
- A model call needs four limits: connect, first byte, idle and total.
- A read or idle timeout never ends a stream that keeps sending data; add a total deadline.
- In httpx combine
httpx.Timeout(...)withasyncio.timeout(...); in Node.js combineAbortSignal.timeout, a re-armed idle timer and the caller's signal withAbortSignal.any. - Cancelling means closing the connection. Whether the server stops generating, and whether you are still billed, depends on the server and on any proxy in between: test it with your own provider.
- Measure real latencies before choosing numbers, and keep the total budget of timeouts plus retries inside what the user will accept.
Sources: httpx timeouts, Python asyncio.timeout, MDN AbortSignal, Node.js globals, RFC 9110, HTTP Semantics.
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/timeouts