Common Crawl

Common Crawl was founded by Gil Elbaz.

The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data. In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and it was not honoring requests from publishers to have their content removed from its databases.

History

Common Crawl was founded in 2007 in San Francisco, and started archiving websites that year. It began publishing its crawls in 2011.

By 2013, sites like TinEye were building their products off of Common Crawl. The crawl reduces the reliance of companies and researchers on Google, which has the biggest dataset. Common Crawl was designed to have more and fresher data that was more efficient to analyze and utilize than the Wayback Machine created by the Internet Archive.

By 2015, 1.8 billion webpages were on the Common Crawl, which started by crawling a list of URLs donated by the search engine Blekko. They use Amazon Web Services, which provides some of its services for free, allowing computing costs to average $2–4000/month. The Common Crawl website listed 30 studies based on Common Crawl data.

Before 2023, Common Crawl was not very well known outside of academic researchers who utilize the data. Common Crawl received its first requests to redact information in 2023 and increasingly started seeing its crawler, CCBot, blocked. In 2023, it began receiving significant financial support from AI companies, including Anthropic and OpenAI, each of which donated $250,000. ChatGPT, as of 2023, was largely trained on Common Crawl, as were other LLMs including Gemini. By April 2023, Common Crawl was capturing 3.1 billion webpages, with an estimated 5% of pages before 2021 containing hate speech or slurs.

As of 2024, Common Crawl had been cited in more than 10,000 academic studies. By 2024, The Pile and Common Crawl had been the two main training datasets being used to train AI models.

In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and when it said that it was honoring requests from publishers to have their content removed from its databases. It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies. As of 2025, Reisner found that CCBot was the most widely-blocked bot by the top 1000 websites.

A 2026 article in LWN.net discussed an advantage to services like Common Crawl being that it can limit the scraping costs to websites by allowing companies and researchers to download the data from Common Crawl instead of scraping it themselves.

In April 2026, Common Crawl experimentally began to distribute its data through Hugging Face Storage Bucket, in addition to its standard storage on Amazon S3.