news-please - an integrated web crawler and information extractor for news that just works

Process Common Crawl data with Python and Spark

News crawling with StormCrawler - stores content as WARC

A very simple news crawler with a funny name

A python utility for downloading Common Crawl data

Price Crawler - Tracking Price Inflation

Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

🕷️ The pipeline for the OSCAR corpus

Statistics of Common Crawl monthly archives mined from URL index files

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

Paskto - Passive Web Scanner

Extract web archive data using Wayback Machine and Common Crawl

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

Index Common Crawl archives in tabular format

Tools to construct and process webgraphs from Common Crawl data

[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

Various Jupyter notebooks about Common Crawl data

CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop