Definition
Adam (Adaptive Moment Estimation) is a popular first-order gradient-based optimization algorithm used in training deep neural networks. It combines the advantages of two other extensions of stochastic gradient descent: AdaGrad, which works well with sparse gradients, and RMSProp, which works well in online and non-stationary settings. Adam maintains exponential moving averages of both the gradient and the squared gradient to adapt the learning rate for each weight individually.
Summary
An optimization algorithm that computes adaptive learning rates for each parameter.
Key Concepts
- Gradient Descent
- Learning Rate
- Momentum
- Variance Estimation
Use Cases
- Deep Learning Training
- Computer Vision Models
- Natural Language Processing
Code Example
| |