Definition
Representation collapse occurs when a neural network, particularly in self-supervised contrastive learning frameworks, learns to map all input data points to the same fixed output vector. This trivial solution minimizes the loss function without learning meaningful features. To prevent this, techniques like normalization, momentum encoders, or specific loss formulations are employed to ensure the model preserves distinct information across different inputs.
Summary
A failure mode in self-supervised learning where the model outputs identical representations for all inputs, losing discriminative power.
Key Concepts
- Self-Supervised Learning
- Contrastive Loss
- Trivial Solutions
- Feature Learning
Use Cases
- Debugging SimCLR or MoCo models
- Improving Contrastive Loss Functions
- Analyzing Model Convergence