Definition
Grokking refers to a counter-intuitive behavior observed in deep learning where a model continues to overfit on training data for a long time, showing poor generalization, before suddenly achieving near-perfect accuracy on both training and test sets. This delayed generalization typically occurs after thousands of epochs, suggesting that the network initially memorizes the data before discovering underlying patterns. It highlights the complex dynamics of optimization landscapes and the relationship between memorization and generalization in neural networks.
Summary
A phenomenon where neural networks suddenly generalize well after prolonged training on small datasets, far beyond the point of memorization.
Key Concepts
- Delayed Generalization
- Overfitting
- Small Datasets
- Optimization Dynamics
Use Cases
- Researching model generalization limits
- Analyzing training dynamics
- Understanding memorization vs. learning