# Token (LLM)

> Source: https://rankxai.com/glossary/token-llm · Last updated: 2026-08-18

A token is the unit a language model processes text in: a fragment that may be a whole word, part of one, a space or a punctuation mark. Tokens are also the unit models are billed and rate-limited by, because they represent the underlying cost of running the model.

## Why is a token not a word?

Because the vocabulary is learned from data rather than from a dictionary. Common words are usually one token; rarer ones split into several, and the same string can tokenise differently depending on the space or punctuation next to it. Numbers, code and non-English text typically use more tokens per character than ordinary English prose does.

The practical version: any rule of thumb converting words to tokens is an approximation, and vendors provide counting endpoints precisely because the approximation is not reliable enough to bill against. It is worth knowing which direction the error runs for you. Text full of product codes, prices, abbreviations or non-English words costs more tokens per visible character than plain prose does, so a specification table is more expensive for a model to carry than the paragraph describing it.

The same is true of markup. A page carrying a great deal of structure relative to its text spends the budget on the structure, which is one of several reasons retrieval pipelines convert HTML to plain text before anything else happens. That conversion is also why JSON-LD contributes nothing to what a model reads: the script tag is removed before the tokens are counted at all.

## Where does the token count actually matter to a publisher?

In two places, and neither is your page directly. The first is the [context window](/glossary/context-window), which is measured in tokens and which everything in a request consumes: the system prompt, the retrieved documents, the tool definitions and the model’s own output. The second is cost, since APIs bill per token in and per token out, which is why any product built on assistants has an incentive to retrieve fewer and shorter passages.

That second incentive is the one worth internalising. The economics of every retrieval system push toward taking less from each source, which is the same direction the measurements point: a page that needs 2,000 words to make its point is more expensive to use than one that needs 200, and there is no counterweight pushing the other way.

The one place a publisher meets tokens directly is in building anything on top of an assistant. A support bot, an internal search or a content tool is billed by them, and the first optimisation anybody makes is retrieving fewer and shorter passages. Every system you are trying to be visible in has that same pressure applied to it by its own economics.

## Do tokens differ between models?

Yes, and it means a token count is not portable. Each model family has its own learned vocabulary, so the same paragraph costs a different number of tokens on different models, and a figure taken from one vendor’s counter does not transfer to another’s billing. For anyone estimating cost, that is a reason to measure against the model you will actually use rather than against a general rule of thumb.

## Sources

- [Anthropic: Messages API reference, on token billing and max_tokens](https://platform.claude.com/docs/en/api/messages), checked 2026-08-18
- [Anthropic: context windows, on what consumes tokens](https://platform.claude.com/docs/en/build-with-claude/context-windows), checked 2026-08-18