Before a language model can do anything with your text, it breaks it into pieces called tokens. This unglamorous first step is invisible to users but explains a surprising number of a model's quirks — and part of what you pay for.
What a token is
A token is a chunk of text — often a word, but frequently a piece of one. Common words might be single tokens; rarer words get split into several ("tokenization" might become "token" + "ization"). The model works entirely in these units: it reads tokens, and it generates tokens. It never sees raw letters or whole words the way we do.
The model doesn't read "strawberry." It reads a couple of tokens that happen to spell it — which is why "how many r's are in strawberry?" trips it up.
Why it explains so much
Tokenization quietly causes several famous behaviors:
- Letter counting fails because the model sees tokens, not individual characters.
- Some languages cost more — languages underrepresented in the tokenizer's training get split into more tokens, so the same sentence uses more tokens (and costs more) than in English.
- Pricing is per token — your bill is measured in these units, so token efficiency is real money.
- Context limits are in tokens, not words — which is why "how long is my document?" depends on how it tokenizes.
Why it matters practically
Understanding tokenization helps you reason about cost (fewer tokens, cheaper), context limits (measured in tokens), and model weaknesses (character-level tasks are hard). It's also an active area — better tokenizers reduce how many tokens text needs, directly cutting cost and stretching context. A humble preprocessing step, but one that shapes how every model reads, what it struggles with, and what it charges.
