Artificial Intelligence
Token
A small unit of text processed by a language model. It may be a word, part of a word or a punctuation mark.
Simple example
“Blue city bike” contains three words, but that alone does not tell you its token count. Different models can divide the words differently.
Why does it matter?
Word count and token count are not interchangeable. Tokens relate to how much text a model can receive and, in some services, how usage is counted. The article’s illustration uses word groups for readability; it does not show actual token boundaries.