Definition
Self-supervised learning is a subset of machine learning where the supervision signal is derived automatically from the data itself, eliminating the need for manual labeling. The model typically solves a pretext task, such as predicting missing words in a sentence or reconstructing masked image patches. This approach leverages vast amounts of unlabeled data to learn robust feature representations, which can then be transferred to various downstream tasks, making it highly scalable and cost-effective for modern foundation models.
Summary
Self-supervised learning is a technique where the model generates its own labels from input data to learn representations without human annotation.
Key Concepts
- Pretext Tasks
- Masked Modeling
- Unlabeled Data
- Representation Learning
Use Cases
- Training BERT via masked language modeling
- Contrastive learning for image embeddings
- Predicting next tokens in LLMs