#crawler

katana
sponsor
katana

A next-generation crawling and spidering framework.

Lookyloo
Lookyloo

Allows users to capture a website page and then display a tree of domains that call each other.

Parsero
Parsero

Robots.txt audit tool.

xurlfind3r
xurlfind3r

A cli utility to find domain's known URLs from curated passive online sources.

smbcrawler
smbcrawler

No-nonsense tool that takes credentials and a list of hosts and crawls through those shares.

socialhunter
socialhunter

Crawls the website and finds broken social media links that can be hijacked

SpiderSuite
SpiderSuite

Advanced web spider/crawler for cyber security professionals.

webpalm
webpalm

A tool that traverses a website and generates a tree of all the webpages and their links.

wildcrawl
wildcrawl

Crawls URL to get a better image of what is tied to a website.

jsfinder
jsfinder

Fetches JavaScript files quickly and comprehensively from a defined list of URLs or domains.

Rock-ON
Rock-ON

All in one recon tool that just get a single domain name and do all of the work alone.

RED HAWK
RED HAWK

All in one tool for information gathering, vulnerability scanning and crawling.

Aranea
Aranea

OSINT tool used for web crawling or examining JavaScript files for likely useful data.

Words Scraper
Words Scraper

Selenium based web scraper to generate passwords list.

dirhunt
dirhunt

Find web directories without bruteforce.

Photon
Photon

Incredibly fast crawler designed for OSINT.

Domain Analyzer
Domain Analyzer

Analyze the security of any domain by finding all the information possible. Made in python.

Web Crawler Security Tool
Web Crawler Security Tool

A web crawler oriented to infosec.

GyoiThon
GyoiThon

Growing penetration test tool using Machine Learning.

SiteBroker
SiteBroker

Utility for information gathering and penetration testing automation.

cariddi
cariddi

Crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.

FestIN
FestIN

The powered S3 bucket finder and content discover.

GitTools
GitTools

A repository with 3 tools for pwn'ing websites with .git repositories available.

gitjacker
gitjacker

Leak git repositories from misconfigured websites.

dvcs-ripper
dvcs-ripper

Rip web accessible version control systems: svn, git...

ScrapeIn
ScrapeIn

Harvest employee email addresses from a specific company through LinkedIn.

urlgrab
urlgrab

A golang utility to spider through a website searching for additional links.

crawley
crawley

The unix-way web crawler.

skipfish
skipfish

Active web application security reconnaissance tool.

CeWL
CeWL

Custom Word List Generator.

ParamSpider
ParamSpider

Mining parameters from dark corners of Web Archives.

gitpillage
gitpillage

Extract data from a .git directory.

gau
gau

Fetch known URLs from several sources.

hakrawler
hakrawler

Simple, fast web crawler designed for discovery of endpoints and assets within a web application.