Definition
Self-attention enables models to capture dependencies between all positions in a sequence simultaneously, regardless of distance. By computing attention scores between every pair of tokens, it allows the network to dynamically focus on relevant context, forming the foundational layer of Transformer architectures used in modern natural language processing and computer vision tasks.
Summary
A mechanism allowing a neural network to weigh the importance of different parts of the input sequence relative to each other.
Key Concepts
- Query-Key-Value
- Attention Scores
- Contextual Weighting
- Parallel Processing
Use Cases
- Machine translation
- Text summarization
- Image classification via Vision Transformers
Code Example
| |