Tor Crawl

TorCrawl.py is an open-source Python-based Tor network web crawler available on GitHub. Developed by MikeMeliz, it is designed to crawl websites through the Tor anonymity network, extracting links, page content, and metadata from both .onion hidden services and clearnet sites accessed via Tor exit nodes.

For OSINT investigators, TorCrawl.py provides automated capability for mapping the structure and content of dark web sites and for accessing clearnet sites through Tor when operational anonymity during investigation is required. The tool's ability to traverse the link graph of a target site systematically makes it useful for comprehensive dark web site analysis.

Primary investigative applications include: mapping the full link structure of a dark web marketplace or forum to understand its scope and architecture, extracting all external references from a target dark web site to identify affiliated infrastructure, archiving page content from dark web sites that may go offline before manual review is complete, and accessing clearnet investigation targets through Tor when operational anonymity is required.

TorCrawl.py provides configurable crawl depth and scope parameters, allowing investigators to limit crawling to specific subdirectories or to expand coverage to linked external sites. This control is important for keeping investigations focused and avoiding unnecessary exposure to out-of-scope content.

The tool outputs structured data including link lists, page content, and metadata, which can be processed with standard data analysis tools for further investigation. The extracted link network can be imported into graph visualization tools to map the relationship structure of a dark web ecosystem.

Technical requirements include Python 3, the Tor service running locally, and the stem library for Tor circuit management. The tool automatically routes all traffic through Tor using the configured SOCKS proxy.

Operational security, legal considerations, and authorization requirements identical to other dark web crawling tools apply. Always operate in an isolated environment, document all crawl parameters and outputs, and verify legal authority before crawling any target system.