Definition
RefinedWeb is a large-scale dataset of filtered web pages designed for pretraining foundation models. It processes billions of web pages to remove low-quality content, duplicates, and harmful material, resulting in a cleaner corpus than raw Common Crawl. This dataset powers the Falcon series of LLMs, demonstrating that high-quality, filtered data can compete with larger, noisier datasets. It emphasizes efficiency and quality in data preparation for generative AI.
Summary
A massive, high-quality web dataset curated by Technology Innovation Institute for pretraining large language models like Falcon.
Key Concepts
- Data Filtering
- Pretraining Corpus
- Web Scraping
- Quality Control
Use Cases
- Pretraining large language models
- Comparing data quality impacts on model performance
- Building efficient training pipelines