What Are Tokens? How AI Breaks Text Into Pieces

When people first use an AI system, they often imagine that it reads and understands text the way humans do — word by word, sentence by sentence. In reality, AI models see text very differently.

At the center of that difference is something called a token.

What is a token?

A token is a small piece of text that an AI model works with internally. It might be a whole word, part of a word, a number, or even a punctuation mark.

AI models do not read letters or words directly. They process sequences of tokens.

For example, a simple sentence might be broken into pieces like this:

• “Artificial”
• “ intelligence”
• “ is”
• “ useful”

The exact breakdown depends on the model, but the idea is the same: text is converted into manageable chunks.

Why AI models use tokens

Tokens make it possible for AI models to handle language mathematically. Each token is represented as a number, which allows the model to calculate probabilities and relationships between pieces of text.

Instead of asking, “What does this sentence mean?”, the model is effectively asking, “Given these tokens, what token is most likely to come next?”

This is why AI models are often described as prediction systems rather than thinking systems.

Tokens and length limits

Every AI model has a limit on how many tokens it can process at once. This includes both the input you provide and the output it generates.

If a conversation or document becomes too long, earlier tokens may be ignored or truncated. This can lead to responses that seem to forget context or miss important details.

Why tokens matter in real use

Understanding tokens helps explain several common AI behaviors:

• Why long prompts sometimes fail
• Why wording changes can affect answers
• Why models may repeat or drift off topic

These behaviors are not signs of confusion or intention. They are side effects of how tokens are processed and predicted.

A useful mental model

One helpful way to think about an AI model is this: it is continuously guessing the next token based on patterns it has seen before. It does this very well — but it is still guessing.

In the next article, we’ll look at how this token-by-token process can sometimes lead to confident but incorrect answers, often described as hallucinations.