Investigator Use
Hakrawler is an open-source web crawler and endpoint discovery tool written in Go, designed for fast, automated discovery of URLs, links, and endpoints within web applications. For penetration testers, bug bounty hunters, and OSINT investigators conducting web reconnaissance, Hakrawler provides rapid JavaScript-aware crawling to map the full surface area of a target web application.
Unlike simple HTTP-based crawlers, Hakrawler uses headless browser rendering to execute JavaScript — critical for discovering endpoints in modern single-page applications (SPAs) built with React, Angular, or Vue, where content and navigation URLs are generated dynamically. This means Hakrawler surfaces URLs that traditional crawlers miss entirely.
Key capabilities include: recursive link following to configurable depth, discovery of JavaScript bundle imports and API endpoints referenced in scripts, extraction of form action URLs, identification of embedded subdomains, and output formatting compatible with other tools in the reconnaissance pipeline (such as Subfinder, Amass, and FFuf).
For bug bounty research, Hakrawler dramatically reduces the manual effort required to map an application's endpoint landscape before beginning vulnerability testing. Running Hakrawler against an in-scope target produces a URL list that can be fed into fuzz testing, parameter enumeration, and vulnerability scanners — accelerating the reconnaissance phase substantially.
For OSINT investigators mapping organizational web presence, Hakrawler can reveal internal-facing paths accidentally exposed, API documentation pages, staging environments, and administrative interfaces that are linked from within the main application but not prominently surfaced in navigation menus.
Installation requires Go: run "go install github.com/hakluke/hakrawler@latest". The tool reads target URLs from stdin, enabling integration with other tools via piping. Common usage: "echo https://example.com | hakrawler -depth 3 -plain".
Limitations include rate limiting and bot detection on sites that actively block automated crawling. Crawling without authorization is illegal and unethical — Hakrawler is for authorized security assessments, bug bounty programs operating within defined scope, or your own applications only.
Document the target, scope authorization, and crawl parameters when using Hakrawler in security assessments.
Before You Pivot
Record Context
Capture the target, search terms, and why this source is relevant before you leave the page.
Preserve Evidence
Archive volatile pages, save screenshots, and keep timestamps for anything that may change.
Corroborate
Treat one tool as a lead source. Confirm important findings with independent sources.
Related Tools
ArchiveBox
Web & URL OSINT
ArchiveBox is self-hosted open-source web archiving for preserving websites, social posts, and online evidence for investigations.
Builtwith
Web & URL OSINT
Web technology information profiler tool. Find out what a website is built with.
Check short url
Web & URL OSINT
CheckShortURL expands shortened URLs to reveal the final destination before clicking, supporting safe analysis of potentially malicious links.
Cute Stats
Web & URL OSINT
Cutestat provides website analytics including traffic estimates, Alexa rank, server details, WHOIS data, and SEO metrics for any domain.
Down for who?
Web & URL OSINT
Down For Everyone Or Just Me confirms whether a website is globally offline or unavailable locally during OSINT investigations.
Fast Osint Crawler
Web & URL OSINT
Photon is a fast OSINT crawler extracting URLs, emails, files, subdomains, and metadata from any target website for investigators.