Token
The unit of text a language model reads and produces, typically a word or a fragment of a word. A model generates its answer one token at a time, each time choosing the next token from a probability distribution, which is the process that sampling settings such as temperature control.
Defined in 3 GAGE programs, which carry 5 distinct definitions of it. The wording above is taught in AI Governance: Applied Mastery.
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
The unit of text a language model reads and produces, typically a word or a fragment of a word. A model generates its answer one token at a time, each time choosing the next token from a probability distribution, which is the process that sampling settings such as temperature control.
The basic unit of text an AI model processes, roughly three-quarters of an English word on average. Model APIs price per million tokens, and input and output tokens are priced differently (output is typically several times the input price).
A fragment of text (often a word, part of a word, or a symbol) that a generative AI processes as a single unit; roughly 0.75 English words per token (OpenAI rule of thumb).
The basic unit of text an AI model processes, roughly three-quarters of an English word on average. Used to measure context window capacity and input or output length.
The basic unit of text processing in language models, roughly three-quarters of a word in English; models have token limits for both input and output.
Where it is taught
The exact lessons this term appears in. The first 7 topics of every program are free with a free account.
- Building Your AI Briefcase · Your AI Learning Companion, AI Literacy & Professional Conduct
- What is AI? Definitions, History, and Types · AI Fundamentals, AI Literacy & Professional Conduct
- How AI Actually Works: Algorithms, Data, and Training · AI Fundamentals, AI Literacy & Professional Conduct
- Prompts, context, and why the same model gives different companies different answers · Build Before You Govern, AI Governance: Applied Mastery
- AI Investment Categories and Cost Structures · The Economics and the Business Case, Business AI Transformation
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.