Counting Tokens Before You Call the API: tiktoken vs Anthropic vs Gemini, and Why the Numbers Differ
Paste the same paragraph into three token counters and you get three different numbers. Each provider ships its own tokenizer, trained on its own corpus with its own vocabulary, so "how many tokens is this prompt" has a different answer for GPT, Claude, and Gemini.
That matters because tokens are what you pay for and what fills your context window. Budget a 200K-context job using a count from the wrong tokenizer and you can be off by enough to blow the window or misprice the run. What you need is not a table of per-model ratios. It is knowing which tool gives you a real count for each provider, and which only gives you an estimate.
This guide covers where the number comes from for each provider, why current Claude models have no local tokenizer at all, and how the Multi-Model Token Comparison tool surfaces the spread across all three at once.
Why one tokenizer disagrees with another
Modern LLM tokenizers are byte-pair encoding (BPE) or a SentencePiece variant. The mechanism is the same in spirit: start from bytes, then merge the most frequent pairs into larger tokens until you reach a fixed vocabulary. The differences come from two places. First, vocabulary size. A tokenizer with a 200,000-token vocabulary can represent common words and code patterns as single tokens that a 100,000-token vocabulary has to split. Second, the training corpus. A tokenizer trained on more code or more multilingual text learns merges that pack that content tighter.
For plain English prose, the major tokenizers usually land close to each other. The gaps widen on content that does not look like English-language web text. Source code, with its braces, snake_case identifiers, and indentation, tokenizes very differently between providers. So does non-Latin script. Arabic, Hindi, Chinese, and Japanese can produce token counts several times higher than the same meaning expressed in English, and the multiplier is not the same across providers.
This is why a per-token price comparison can mislead. The cheaper per-token model is not always cheaper per request. If model A charges less per token but its tokenizer splits your Hindi support tickets into 30% more tokens than model B, model B can win on the actual bill. You only see that by counting your real content through each tokenizer, which is what the comparison tool is for.
OpenAI: tiktoken gives you an exact count locally
OpenAI open-sources its tokenizer as tiktoken, so you can count tokens on your own machine with zero API calls, zero latency, and no key. The work is in picking the right encoding for the model you are actually calling.
o200k_base— the GPT-4o family, GPT-5 family, and the o-series reasoning models.cl100k_base— GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo.
Pick the wrong encoding and your count is off, because these two vocabularies tokenize the same text differently. In Python it is a few lines:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("Count these tokens before you ship.")
print(len(tokens))One trap: this counts the raw string. A real chat request wraps every message in role and formatting tokens, so a multi-message request costs a few tokens more per message than the sum of its content. For budgeting a context window, a per-message overhead of a handful of tokens is usually noise. For tight per-call cost accounting on short messages, account for it. OpenAI documents the exact framing for its chat format; if you need the number to the token, follow that counting guidance rather than raw encode.
Anthropic: there is no local tokenizer for current Claude
Moving a pipeline from OpenAI to Claude, the offline trick has no equivalent. There is no supported local tokenizer for current Claude models. Older Claude generations shipped a tokenizer you could run offline, but it does not match how the current models count, and Anthropic does not publish a local tokenizer for them.
Reaching for tiktoken as a substitute is the common mistake, and it is wrong for Claude. It is OpenAI's tokenizer with OpenAI's vocabulary. It tends to undercount Claude tokens on ordinary text and miss by more on code and non-English input. A number from tiktoken, gpt-tokenizer, or any GPT-calibrated estimator is not a Claude token count.
The authoritative number comes from Anthropic's count-tokens endpoint, POST /v1/messages/count_tokens. It is a real API call, so it needs a key and a round trip, but it returns the count Anthropic's own context-window logic uses, computed with the model's actual tokenizer. Anthropic notes one caveat: it treats the result as an estimate that can differ by a small amount and may include system-added tokens you are not billed for. Trust it over any local guess. Because it takes messages rather than a bare string, the count already includes role and message framing.
from anthropic import Anthropic
client = Anthropic()
resp = client.messages.count_tokens(
model="claude-opus-4-8",
messages=[{"role": "user", "content": open("prompt.txt").read()}],
)
print(resp.input_tokens)Counts are model-specific, so pass the same model ID you will use for inference. Different Claude generations can tokenize the same text differently; do not assume a count from one carries to another.
Gemini: count through the API, not a guess
Gemini has no first-party offline tokenizer you drop into a script the way you do with tiktoken, but the Gemini API exposes a countTokens method, so you can get an authoritative count for a given model before you send the real request.
The same rule applies: the number is model-specific, and you should count against the model you intend to call. Any browser-side number you see for Gemini without an API call is a calibrated approximation, not the value Google's backend will charge. For rough budgeting that is often fine. For anything where being over the context limit breaks the job, confirm with countTokens.
Exact vs estimated, and when each is enough
You do not always need the authoritative number. Sort your decisions into two buckets.
An estimate is fine when you are sketching whether a document family fits a context window, comparing tokenizer efficiency across providers to pick a model, or showing an approximate cost in a UI. Being within a few percent does not change the decision.
You want the provider's count when a request sits near a hard context limit, you are reconciling a bill, or you are enforcing a per-request token cap in production. Here a 15% miss is the difference between a request that runs and one the API rejects.
That is the line the Multi-Model Token Comparison draws. It runs real tiktoken for the OpenAI models, so those counts are exact and labeled by encoding (o200k_base or cl100k_base). For Claude and Gemini it shows a calibrated estimate, marked as such, because there is no local tokenizer to give an exact answer in the browser. You see the spread instantly, and you know which numbers to trust as-is and which to confirm against the provider's count endpoint before you commit.
A workflow that holds up later
Put it together into something repeatable.
- Prototype with the comparison tool. Paste a representative sample of your real content, not a clean English paragraph. If your traffic is code or Hindi, paste code or Hindi. Read the spread across providers and note where the per-token price ranking flips once tokenization is counted in.
- Lock in OpenAI counts with
tiktoken. Match the encoding to the model, and add per-message framing if you are accounting to the token. - Confirm Claude with the count endpoint. No local shortcut exists;
count_tokenswith the exact model ID is the number to trust. Do not substitute a GPT tokenizer. - Confirm Gemini with
countTokens. Same discipline, same model-specific caveat. - Re-baseline when you change models. A new generation can shift counts even within one provider. Anthropic introduced a new tokenizer with Claude Opus 4.7 that produces roughly 30% more tokens than earlier Claude models for the same text, and current models use it, so a count measured on an older Claude does not carry over.
Once you have real counts, feed them into the rest of the budget. The LLM Cost Calculator and Prompt Token Cost turn a token count into a per-call price, and the Context Window Comparison tells you which models can hold the request at all. The token count is the input to all of it, which is why getting it from the right tokenizer is the first thing to nail down.
Skip the manual work. The companion tool runs this in your browser, with nothing uploaded.
Multi-Model Token ComparisonFrequently asked questions
Can I use tiktoken to count Claude or Gemini tokens?
No. tiktoken is OpenAI's tokenizer with OpenAI's vocabulary. It tends to undercount Claude tokens on ordinary text and miss by more on code and non-English input, and it is not calibrated for Gemini either. For Claude, use Anthropic's count-tokens endpoint; for Gemini, use the API's countTokens method.
Why do OpenAI, Claude, and Gemini report different token counts for the same text?
Each provider trains its own tokenizer on its own corpus with its own vocabulary size. Different vocabularies merge characters into tokens differently, so the same string splits into a different number of tokens. The gaps are usually small for English prose and largest for source code and non-Latin scripts.
Is there a local tokenizer for current Claude models?
No supported one. Older Claude generations had an offline tokenizer, but it does not match how current models count, and Anthropic does not publish a local tokenizer for them. The authoritative count comes from the POST /v1/messages/count_tokens endpoint, which is a real API call. Anthropic treats the returned number as an estimate that may differ by a small amount.
Does counting the raw text match what the API actually charges?
Not exactly. A chat request wraps each message in role and formatting tokens, so a real multi-message request costs a few tokens more per message than the sum of the content. For context-window budgeting that overhead is usually noise; for tight per-call accounting, include it. Anthropic's count endpoint already accounts for framing because it takes full messages, not a bare string.
Which OpenAI encoding should I use, o200k_base or cl100k_base?
Use o200k_base for the GPT-4o family, GPT-5 family, and the o-series reasoning models. Use cl100k_base for GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo. The two vocabularies tokenize the same text differently, so matching the encoding to the model matters.
When is an estimated token count good enough?
Estimates are fine for rough context-window planning, comparing tokenizer efficiency to pick a model, or showing approximate cost in a UI. Get the provider's count when a request is near a hard context limit, when you are reconciling a bill, or when you enforce a per-request token cap in production.