Definition
Layer Normalization stabilizes training by reducing internal covariate shift, particularly effective in recurrent and transformer architectures. Unlike Batch Normalization, which depends on batch statistics, Layer Normalization computes mean and variance across all features of a single training example. This makes it robust to small batch sizes and sequential data processing, leading to faster convergence and improved model stability.
Summary
A technique that normalizes the activations of a neural network layer across the feature dimension for each individual sample.
Key Concepts
- Normalization
- Internal Covariate Shift
- Transformer Models
- RNNs
Use Cases
- Training Transformer models like BERT
- Stabilizing RNN/LSTM training
- Deep learning with small batch sizes
Code Example
| |