Definition
Tokens are the fundamental building blocks of input data in NLP, typically representing words, subwords, or characters. Large Language Models (LLMs) process text by converting it into tokens, which are then mapped to numerical vectors. The way text is tokenized significantly impacts model performance, context window size, and computational efficiency. Tokens allow models to handle variable-length inputs and capture semantic meaning at a granular level, forming the basis for understanding and generating language.
Summary
A token is a discrete unit of text or data that serves as the basic input element for natural language processing models.
Key Concepts
- Discrete Unit
- Vocabulary
- Embedding
- Context Window
Use Cases
- Text generation in LLMs
- Machine translation preprocessing
- Sentiment analysis input formatting