Definition
In the context of optimization, divergence occurs when the parameters of a model update in a way that causes the loss to increase rather than decrease, often leading to NaN values or infinite gradients. This is frequently caused by excessively high learning rates, poor weight initialization, or numerical instability in the computation graph. Detecting divergence early is crucial for debugging training pipelines, as it prevents wasted computational resources and ensures the model can converge to a meaningful solution. Techniques like gradient clipping or reducing the learning rate are common remedies.
Summary
Divergence refers to the failure of a machine learning algorithm’s loss function to decrease during training, resulting in unstable or worsening performance.
Key Concepts
- Loss Explosion
- Learning Rate Sensitivity
- Gradient Instability
- Numerical Precision
Use Cases
- Debugging training loops in deep learning frameworks
- Tuning hyperparameters for stable convergence
- Implementing gradient clipping strategies