Multi-Head Attention - AI Terms Dictionary

Definition

Multi-Head Attention extends the standard attention mechanism by running it multiple times in parallel with different learned linear projections. This enables the model to jointly attend to information from different positional subspaces at different positions. By capturing diverse relationships within the input sequence, such as syntactic and semantic dependencies, it significantly enhances the model’s ability to understand context. It is a foundational component of modern Large Language Models (LLMs) and vision transformers, providing robust feature extraction capabilities.

Summary

A mechanism in transformer models that allows the model to attend to information from different representation subspaces simultaneously.

Key Concepts

Self-Attention
Linear Projections
Concatenation

Use Cases

Natural Language Processing (NLP)
Machine Translation
Image Classification with Vision Transformers

Code Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, x):
        # Simplified forward pass logic
        pass

Definition

Summary

Key Concepts

Use Cases

Code Example

Related Terms