Web Crawl Data

Common Crawl is a non-profit organization that produces and maintains an open repository of web crawl data containing petabytes of web content accumulated since 2008. For data scientists, OSINT researchers, and computational analysts, Common Crawl provides raw access to the full text and metadata of billions of crawled web pages — enabling large-scale analysis of the public internet's content at a scale impossible to replicate independently.

The Common Crawl corpus is released monthly and made available through Amazon S3, allowing researchers to access the data programmatically without any licensing restrictions. Each monthly release contains WARC (Web ARChive) files with raw HTML, WAT files with metadata, and WET files with extracted plain text — supporting different analysis approaches from full content analysis to lightweight metadata-only queries.

For OSINT applications, Common Crawl enables investigators to search for historical appearances of specific content, domains, or entities across the archived web. Because the corpus spans more than a decade, researchers can find evidence of content that has since been deleted — discovering when specific information first appeared online, tracking how an organization's messaging evolved over time, or finding cached versions of pages removed from search engine indexes.

Computational OSINT workflows using Common Crawl include: identifying all domains registered to a specific organization over time, tracking the appearance of specific phone numbers, email addresses, or names across web content, monitoring changes to key pages, and analyzing link structures between domains. These analyses require programming skills and significant compute resources, typically executed on AWS using Athena or EMR to query the S3-hosted data efficiently.

The Index (the CC-Index) provides a searchable index of all URLs in the corpus, allowing investigators to check whether a specific URL was crawled without downloading the full dataset.

For lighter-weight access to similar historical web data, consider Wayback Machine CDX API as a complementary resource.