Token (LLM)

Why is a token not a word?

Because the vocabulary is learned from data rather than from a dictionary. Common words are usually one token; rarer ones split into several, and the same string can tokenise differently depending on the space or punctuation next to it. Numbers, code and non-English text typically use more tokens per character than ordinary English prose does.

The practical version: any rule of thumb converting words to tokens is an approximation, and vendors provide counting endpoints precisely because the approximation is not reliable enough to bill against. It is worth knowing which direction the error runs for you. Text full of product codes, prices, abbreviations or non-English words costs more tokens per visible character than plain prose does, so a specification table is more expensive for a model to carry than the paragraph describing it.

The same is true of markup. A page carrying a great deal of structure relative to its text spends the budget on the structure, which is one of several reasons retrieval pipelines convert HTML to plain text before anything else happens. That conversion is also why JSON-LD contributes nothing to what a model reads: the script tag is removed before the tokens are counted at all.

Where does the token count actually matter to a publisher?

In two places, and neither is your page directly. The first is the context window, which is measured in tokens and which everything in a request consumes: the system prompt, the retrieved documents, the tool definitions and the model’s own output. The second is cost, since APIs bill per token in and per token out, which is why any product built on assistants has an incentive to retrieve fewer and shorter passages.

That second incentive is the one worth internalising. The economics of every retrieval system push toward taking less from each source, which is the same direction the measurements point: a page that needs 2,000 words to make its point is more expensive to use than one that needs 200, and there is no counterweight pushing the other way.

The one place a publisher meets tokens directly is in building anything on top of an assistant. A support bot, an internal search or a content tool is billed by them, and the first optimisation anybody makes is retrieving fewer and shorter passages. Every system you are trying to be visible in has that same pressure applied to it by its own economics.

Do tokens differ between models?

Yes, and it means a token count is not portable. Each model family has its own learned vocabulary, so the same paragraph costs a different number of tokens on different models, and a figure taken from one vendor’s counter does not transfer to another’s billing. For anyone estimating cost, that is a reason to measure against the model you will actually use rather than against a general rule of thumb.

Related terms

• Context WindowA context window is all the text a language model can reference when generating a response, including the response itself.
• Temperature (LLM)Temperature is the parameter controlling how much randomness a language model injects into its response.
• ChunkingChunking is the step that splits a page into passages so a retrieval system can select one without the rest.
• EmbeddingAn embedding is a list of numbers representing a piece of text, positioned so that texts with similar meaning sit close together.