Token

What is a Token?

A token is the smallest piece of text that an AI language model reads and processes. Think of it like breaking a sentence into individual puzzle pieces. Sometimes a token is a whole word, sometimes just part of a word, and sometimes even a single character or punctuation mark.

For example, the word "unbelievable" might be split into three tokens: "un", "believ", and "able". Short common words like "the" or "cat" are typically one token each. This splitting process is handled by a component called a tokenizer.

How Does It Work?

Before any AI model can understand your text, the tokenizer converts your sentence into a sequence of tokens, each mapped to a number. The model then processes these numbers rather than raw text. Different models use different tokenization strategies. GPT-style models use a method called Byte Pair Encoding (BPE), which learns the most efficient way to split text based on how frequently character combinations appear in training data.

Why Does It Matter?

Tokens directly affect cost and performance. Most AI APIs charge per token, so understanding tokenization helps you estimate expenses. Models also have a maximum token limit (called a context window), which determines how much text you can send and receive in a single interaction. Longer prompts consume more tokens, leaving less room for the model's response. Efficient prompt writing often means being mindful of your token budget.