Count tokens and estimate costs before you call a model (Python and JavaScript)
Three ways to count tokens (local tokenizer, provider endpoint, usage), how to turn them into a cost range, and a budget check that blocks expensive calls.

Model APIs bill by the token, and the bill arrives after the work is done. A prompt that pulls in a long document, a chat history that grows without limit, or a loop that calls the model more often than planned can all turn a cheap feature into an expensive one before anyone notices. Counting tokens before the call lets you do three useful things: refuse requests that are over budget, check that the prompt fits the model's context window, and send small jobs to a cheaper model.
This guide shows how to count tokens in Python and JavaScript, how to turn the count into a cost range, and how to check the real cost after the call. Versions checked on 30 October 2026: tiktoken 0.14.0, anthropic 2.3.0 and openai 3.41.0 on PyPI, js-tiktoken 1.0.21 on npm.
A token is not a word, and not the same everywhere
Models read text as tokens: pieces of words, whole short words, punctuation, bytes of unusual characters. Each model family has its own tokenizer, so the same text gives different counts on different models. Three sentences from our test, counted with OpenAI's o200k_base and cl100k_base encodings:
| Text | Characters | Words | o200k_base | cl100k_base |
|---|---|---|---|---|
| English sentence | 88 | 16 | 18 | 18 |
| The same sentence in Italian | 90 | 14 | 23 | 26 |
| Two lines of Python with an emoji | 54 | 13 | 21 | 22 |
The Italian version has fewer words and about the same number of characters, but it needs five more tokens with o200k_base and eight more with the older encoding. Rules of thumb such as "four characters per token" are calibrated on English text; for other languages and for code, measure.
The tokenizer can also change within the same provider. Anthropic's pricing page states that Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text. A count measured on one model is not valid for another: count again when you change model.
Three ways to count
| Method | Where it runs | What it covers | When to use it |
|---|---|---|---|
Local tokenizer (tiktoken, js-tiktoken) |
Your process, offline | Plain text, OpenAI encodings only | Fast checks, trimming text to a size, many calls per second |
| Provider counting endpoint | Provider's API | The whole request: system prompt, messages, tools, images | Before an expensive call, when you need the real input count |
usage in the response |
After the call | What you were actually billed for | Cost tracking, alerts, checking your estimates |
For Claude models, Anthropic's documentation points to its counting endpoint rather than a local tokenizer.
Count locally in Python with tiktoken
import tiktoken
def count_tiktoken(text: str, encoding: str = "o200k_base") -> int:
"""Local count for OpenAI models (o200k_base: GPT-4o, GPT-4.1, GPT-5, o-series). Text only."""
return len(tiktoken.get_encoding(encoding).encode(text))
tiktoken.encoding_for_model("gpt-4o") returns the right encoding for a model name, but only for the names in the library's own table; for a model it does not know it raises a KeyError, so for new models it is simpler to choose the encoding yourself. In tiktoken 0.14.0 the table maps GPT-4o, GPT-4.1, GPT-5 and the o-series to o200k_base.
Two practical notes. The first call downloads the encoding file from OpenAI's public storage and caches it; in a container without internet access, set TIKTOKEN_CACHE_DIR and ship the file with the image. And the count covers text only: the API also counts the message structure, tools and images, which the text alone does not include. Treat a local count of the messages' text as a lower bound.
Count locally in JavaScript with js-tiktoken
js-tiktoken is a pure JavaScript port that works in Node.js and in bundlers. Its lite entry point lets you load only the encoding you need, which keeps the bundle small:
import { Tiktoken } from 'js-tiktoken/lite';
import o200k_base from 'js-tiktoken/ranks/o200k_base';
const enc = new Tiktoken(o200k_base); // load once: building the encoder is the slow part
export function countTiktoken(text) {
return enc.encode(text).length;
}
Our test runs the Python and JavaScript versions on the same texts and checks that the counts are identical, so you can count in the browser to warn the user and count again on the server to enforce the limit.
Ask the provider for the exact count
Anthropic has a token counting endpoint, POST /v1/messages/count_tokens. It accepts the same input as a Messages request, including the system prompt, tools, images and PDFs, and returns input_tokens. According to the documentation it is free, has its own requests-per-minute limit separate from message creation, and returns an estimate that can differ from the real usage "by a small amount". It does not accept server tools, the MCP connector, or images and documents given by URL or file reference.
OpenAI offers a similar endpoint for the Responses API, POST /v1/responses/input_tokens. With the official Python SDKs:
def count_claude(client, model: str, messages: list, system: str | None = None) -> int:
"""Exact input count from Anthropic's free count_tokens endpoint (includes system prompt and tools)."""
extra = {"system": system} if system else {}
return client.messages.count_tokens(model=model, messages=messages, **extra).input_tokens
def count_openai(client, model: str, text: str) -> int:
"""Input count from OpenAI's POST /v1/responses/input_tokens endpoint."""
return client.responses.input_tokens.count(model=model, input=text).input_tokens
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
n = count_claude(client, "claude-sonnet-5-5", [{"role": "user", "content": "Summarise this contract: ..."}])
Without an SDK, the call is a plain HTTP request. These are the headers shown in Anthropic's documentation; run this on your server, never in the browser, because it needs your API key:
export async function countClaudeTokens({ model, messages, system, apiKey, baseUrl = 'https://api.anthropic.com' }) {
const res = await fetch(`${baseUrl}/v1/messages/count_tokens`, {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' },
body: JSON.stringify({ model, messages, ...(system ? { system } : {}) }),
});
if (!res.ok) throw new Error(`count_tokens HTTP ${res.status}: ${await res.text()}`);
return (await res.json()).input_tokens;
}
The counting endpoint is an HTTP call with its own rate limit, so it can answer 429 like any other API. Our retry and backoff simulator and the guide on HTTP 429 and Retry-After apply here too.
From tokens to money
Prices are quoted in dollars per million tokens (MTok), with separate prices for input and output. On 30 October 2026 Anthropic's pricing page listed, among others:
| Model | Input | Output | Cache write (5 min) | Cache read |
|---|---|---|---|---|
| Claude Sonnet 5.5 | $2 / MTok | $10 / MTok | $2.50 / MTok | $0.10 / MTok |
| Claude Opus 5.5 | $4 / MTok | $20 / MTok | $5 / MTok | $0.20 / MTok |
Prices change, so keep them in configuration and check the provider's page; do not trust a copy in code or in an article, this one included. Several modifiers apply on top: the Batch API halves input and output prices, some models price by prompt length (Claude Haiku 5.5 costs more for prompts over 100,000 tokens), and US-only inference adds 10% on Claude 4.6 and later models.
Before the call you know the input tokens but not the output. You do know the maximum: the max_tokens you send. Anthropic's extended thinking documentation states that max_tokens remains the hard ceiling on total output, thinking included, and that thinking tokens are billed as output. So the honest estimate is a range:
from dataclasses import dataclass
@dataclass(frozen=True)
class Price:
"""USD per million tokens (MTok)."""
input: float
output: float
cache_write: float = 0.0 # 5-minute cache write
cache_read: float = 0.0
def estimate_cost(input_tokens: int, max_output_tokens: int, price: Price) -> tuple[float, float]:
"""Cost range in USD: from an empty answer to an answer that uses all of max_output_tokens."""
low = input_tokens * price.input / 1_000_000
high = low + max_output_tokens * price.output / 1_000_000
return low, high
A request with 12,000 input tokens and max_tokens of 1,000 costs between $0.024 and $0.034 on Claude Sonnet 5.5, and between $0.048 and $0.068 on Claude Opus 5.5. The difference looks tiny until you multiply it: 50,000 such calls a month is up to $1,700 on Sonnet 5.5 and up to $3,400 on Opus 5.5. Output tokens cost five times as much as input tokens on both models, so a lower max_tokens is often the cheapest saving you can make.
Block calls that are over budget
With a count and a price, a guard is three lines. Use the worst case, not the average:
class BudgetExceeded(Exception):
pass
def check_budget(input_tokens: int, max_output_tokens: int, price: Price, max_usd: float) -> float:
"""Refuse a call whose worst case costs more than max_usd. Returns the worst case."""
_, high = estimate_cost(input_tokens, max_output_tokens, price)
if high > max_usd:
raise BudgetExceeded(f"worst case ${high:.4f} is over the ${max_usd:.4f} limit")
return high
What you do when the check fails depends on the feature: shorten the context (drop old chat turns, retrieve fewer documents), switch to a cheaper model, lower max_tokens, or tell the user that the document is too long. All of these are better than discovering the cost on the invoice.
Check the real cost after the call
Every response reports what was used. For Claude, the usage object contains input_tokens, output_tokens, cache_creation_input_tokens and cache_read_input_tokens. One detail trips people up: according to the prompt caching documentation, input_tokens counts only the tokens after the last cache breakpoint. The total input is the sum of the three input fields, each with its own price:
def actual_cost(usage, price: Price) -> float:
"""Cost of a finished Claude call from response.usage (input_tokens excludes cached tokens)."""
return (usage.input_tokens * price.input
+ (usage.cache_creation_input_tokens or 0) * price.cache_write
+ (usage.cache_read_input_tokens or 0) * price.cache_read
+ usage.output_tokens * price.output) / 1_000_000
In OpenAI's Responses API, usage has input_tokens, output_tokens and total_tokens, and output_tokens_details.reasoning_tokens tells you how many output tokens were hidden reasoning. Local models report usage too: Ollama returns prompt_eval_count and eval_count in its final message.
Log these numbers with the feature name and the model for every call. Comparing them with your estimates is the only way to find out whether the estimates are any good, and a daily sum per feature is the simplest cost alert you can build. To estimate how many concurrent calls your peak traffic means for the backend, our API capacity sizing tool does the arithmetic.
Test it without an API key
The example folder has a small fake server, written with the Python standard library, that answers the two counting endpoints with fixed numbers and records each request. The tests point the official anthropic and openai SDKs and the fetch version at it, so they check the URL, headers and body your code sends without spending anything. The other tests compare Python and JavaScript tokenizer counts and check the cost arithmetic, including cache reads:
test_actual_cost_with_cache ... ok
test_count_claude_with_fetch ... ok
test_count_claude_with_sdk ... ok
test_count_openai_with_sdk ... ok
test_estimate_and_budget ... ok
test_python_and_javascript_tokenizers_agree ... ok
The suite runs in GitHub Actions on every change.
Limits
- A local count is exact only for the text and only for the encoding you chose. For chat requests, tools and images, it is a lower bound.
- The provider counting endpoints are estimates too, close but not guaranteed to match the bill.
- Output length is the part you cannot know in advance. Agents and tool loops make it worse, because each step is a new call with a longer context: put the budget check inside the loop, not only before it.
- The prices in this article and in the repository are a snapshot taken on 30 October 2026.
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/token-costs