Common Crawl
Common Crawl is a nonprofit 501 organization that crawls the web and freely provides its archives and datasets to the public.
Common Crawl was founded by Gil Elbaz. It is funded by the Elbaz Family Foundation Trust and significant donations from the AI industry.
Contents archived by Common Crawl are mirrored and made available online in the Wayback Machine. They are used by researchers, as well as AI companies to train large language models.
In November 2025, an investigation by The Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases.
History
Advisors to the non-profit have included Peter Norvig and Joi Ito.By 2013, sites like TinEye were building their products off of Common Crawl.
As of 2016, the Common Crawl dataset includes copyrighted work and is distributed from the US under fair use claims. Researchers in other countries have made use of techniques such as shuffling sentences or referencing the Common Crawl dataset to work around copyright law in other legal jurisdictions.
A filtered version of Common Crawl was used to train OpenAI's GPT-3 language model, announced in 2020. In 2023, it began receiving significant financial support from AI companies, including Anthropic and OpenAI, each of which donated $250,000.
As of 2024, Common Crawl had been cited in more than 10,000 academic studies.
In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases. It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies.