Definition
Batch size is a critical hyperparameter that determines how many samples are processed before the model’s internal parameters are updated. A larger batch size provides a more accurate estimate of the gradient, leading to stable convergence but requiring more memory and potentially generalizing poorly. Conversely, smaller batch sizes introduce noise into the gradient estimation, which can help escape local minima but may result in noisier convergence paths and longer training times due to frequent updates.
Summary
The number of training examples utilized in one iteration of the stochastic gradient descent algorithm.
Key Concepts
- Gradient Estimation
- Memory Constraints
- Convergence Stability
- Hyperparameter Tuning
Use Cases
- Tuning model convergence speed
- Managing GPU memory limits during training
- Improving generalization via noisy gradients