Token, in the context of artificial intelligence, is the basic unit of text that a language model processes and generates. Think of it as breaking down a sentence into digestible chunks, much like how you might break a word into syllables when teaching a child to read. While we naturally read and write in complete words and sentences, AI models work more efficiently by splitting text into smaller pieces called tokens.

Similarly, when an AI model processes your input or generates a response, it doesn’t work with whole words as we might expect. Instead, it breaks the text into tokens, which can be whole words, parts of words, or even individual characters, depending on how common they are. For example, the word “understanding” might be split into tokens like “under,” “stand,” and “ing,” while a common word like “the” would be a single token. Punctuation marks and spaces are also typically treated as separate tokens.

During text processing, the AI model converts these tokens into numerical representations that it can mathematically manipulate and analyze. Every token the model reads (your input) and every token it generates (its output) counts toward the model’s processing capacity. This is why AI services often measure usage in tokens rather than words or characters—it’s the actual unit of computation the model works with.

The result is a system that can efficiently handle diverse languages, rare words, and even misspellings by breaking them into recognizable token patterns. Understanding tokens is also important for practical reasons: token limits determine how much text you can include in a single conversation, and token counting affects the cost and speed of using AI models.