How LLM Tokenization Works (and Why It Costs)

Large language models don't read words. They read tokens. Every prompt you send and every reply you get back is counted, billed, and capped in tokens, not in characters or words. If you've been surprised by an API bill, or hit a "context length exceeded" error on a prompt that looked short, tokenization is usually the reason.

This post covers what a token actually is, why the count is hard to predict, and how to estimate it before you ship. You can check any text against the LLM Token Counter as you read.

What is a token?

A token is a chunk of text, usually a few characters, produced by the model's tokenizer. Tokenizers are built with algorithms like Byte Pair Encoding (BPE). They start from individual characters and keep merging the most common pairs until they have a fixed vocabulary of maybe 100,000 sub-word units.

In practice that means common words are often a single token, while rare words get split into pieces. "token" might be one token, but "tokenization" can split into token plus ization. A leading space usually counts as part of the token too, so " the" and "the" can tokenize differently.

The rule of thumb for English prose is about 1 token to ¾ of a word, or roughly 4 characters per token. So 1,000 tokens is around 750 words. That's fine for a first guess. It's also wrong for a lot of real input.

Why "4 characters per token" breaks

The ¾-of-a-word rule assumes ordinary English. As soon as your text stops looking like prose, the ratio shifts, and usually not in your favour:

Code and JSON. Brackets, indentation, quotes and operators fragment into lots of small tokens. A minified JSON blob can use far more tokens per character than the same number of words of English.
Other languages. Text in a language the tokenizer wasn't tuned for, or any non-Latin script, often takes several tokens per character.
Emoji and symbols. A single emoji can be two or more tokens. Maths symbols, box-drawing characters and unusual Unicode all push the count up.
Long numbers and IDs. A UUID or a long number is rarely one token. It gets chopped into several.

This is why you can't eyeball a token count. Two prompts with the same word count can differ by 2x in token count depending on what's in them.

Different models count differently

There's no universal token. Each model family ships its own tokenizer, so the same text produces different counts on different models. A prompt that's 1,000 tokens for one model might be 1,150 for another. When you switch models, even within the same provider, measure again rather than assuming the old number holds.

One trap worth naming: don't reach for a generic library like tiktoken to count tokens for a non-OpenAI model. It's OpenAI's tokenizer, and it will undercount other models' tokens, sometimes badly on code or non-English text. Use the provider's own counter, or a general estimate (like the one on this site) when you're budgeting.

Input tokens, output tokens, and the bill

You pay for two things, usually at different rates.

Input tokens: everything you send. The system prompt, the conversation history, retrieved documents, and the user's message.
Output tokens: everything the model generates back.

Output is normally the more expensive of the two. As of mid-2026, a typical frontier model runs roughly $5 per million input tokens and $25 per million output tokens, so output costs about five times as much. (Check current pricing; these numbers move, and vary by provider.)

The cost driver people forget is history. APIs are stateless, so every turn of a chat re-sends the whole conversation as input. A 2,000-token system prompt plus a growing transcript gets paid for again on every request. Ten turns in, most of your input bill is just re-reading what you already sent.

A worked example

Say you're running a support assistant:

System prompt: about 1,500 tokens
Average user message: about 200 tokens
Average reply: about 400 tokens

For one request that's roughly 1,700 input tokens and 400 output tokens. At $5 and $25 per million:

Input: 1,700 × $5 / 1,000,000 ≈ $0.0085
Output: 400 × $25 / 1,000,000 = $0.0100
Total: about $0.0185 per request

Under two cents. But run 100,000 requests a month and you're at roughly $1,850, before history starts growing the input on longer chats. Tokens are cheap one at a time and expensive in bulk, which is the whole reason to estimate them up front.

Five ways to cut token usage

Trim the system prompt. You pay for it on every request, so tighten it and move rarely-needed detail into retrieval.
Summarize or truncate history. Don't resend the entire transcript forever. Compact older turns once a conversation gets long.
Ask for structured, bounded output. "Reply with a JSON object of at most three fields" produces fewer output tokens than an open-ended essay.
Cache repeated context. If you send the same large preamble every time, prompt caching can cut the cost of that prefix a lot.
Measure before you ship. Paste a representative prompt into the token counter and price it at your model's rate before it's running at scale.

So what?

Tokens are the real currency of LLM APIs, and they don't map neatly to words. Code, other languages, emoji and long IDs all cost more than they look. History quietly compounds your input bill. And every model counts differently. The fix is boring but it works: measure. Paste your prompts into the LLM Token Counter, price them out, and the invoice stops being a surprise.