AI Terms Dictionary

A comprehensive multilingual AI terminology dictionary

Definition

Since transformers process all tokens in parallel rather than sequentially like RNNs, they lack inherent knowledge of token order. Positional encoding adds specific vectors to input embeddings to preserve sequence information. Common methods include sinusoidal functions learned during training or learned embeddings. This allows the self-attention mechanism to weigh the importance of different tokens based on their position, enabling the model to understand syntax and context effectively.

Summary

A technique that injects information about the relative or absolute position of tokens in a sequence into transformer models.

Key Concepts

Use Cases

Code Example

1
2
3
4
5
6
7
8
9

import torch
import math
def get_positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe.unsqueeze(0)