#crawler

katana on offsec.tools
Sponsor
katana

A next-generation crawling and spidering framework.

xurlfind3r on offsec.tools
xurlfind3r

A cli utility to find domain's known URLs from curated passive online sources.

smbcrawler on offsec.tools
smbcrawler

No-nonsense tool that takes credentials and a list of hosts and crawls through those shares.

socialhunter on offsec.tools
socialhunter

Crawls the website and finds broken social media links that can be hijacked

SpiderSuite on offsec.tools
SpiderSuite

Advanced web spider/crawler for cyber security professionals.

webpalm on offsec.tools
webpalm

A tool that traverses a website and generates a tree of all the webpages and their links.

wildcrawl on offsec.tools
wildcrawl

Crawls URL to get a better image of what is tied to a website.

jsfinder on offsec.tools
jsfinder

Fetches JavaScript files quickly and comprehensively from a defined list of URLs or domains.

Rock-ON on offsec.tools
Rock-ON

All in one recon tool that just get a single domain name and do all of the work alone.

RED HAWK on offsec.tools
RED HAWK

All in one tool for information gathering, vulnerability scanning and crawling.

Aranea on offsec.tools
Aranea

OSINT tool used for web crawling or examining JavaScript files for likely useful data.

Words Scraper on offsec.tools
Words Scraper

Selenium based web scraper to generate passwords list.

dirhunt on offsec.tools
dirhunt

Find web directories without bruteforce.

Photon on offsec.tools
Photon

Incredibly fast crawler designed for OSINT.

Domain Analyzer on offsec.tools
Domain Analyzer

Analyze the security of any domain by finding all the information possible. Made in python.

Web Crawler Security Tool on offsec.tools
Web Crawler Security Tool

A web crawler oriented to infosec.

GyoiThon on offsec.tools
GyoiThon

Growing penetration test tool using Machine Learning.

SiteBroker on offsec.tools
SiteBroker

Utility for information gathering and penetration testing automation.

cariddi on offsec.tools
cariddi

Crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.

FestIN on offsec.tools
FestIN

The powered S3 bucket finder and content discover.

GitTools on offsec.tools
GitTools

A repository with 3 tools for pwn'ing websites with .git repositories available.

gitjacker on offsec.tools
gitjacker

Leak git repositories from misconfigured websites.

dvcs-ripper on offsec.tools
dvcs-ripper

Rip web accessible version control systems: svn, git...

ScrapeIn on offsec.tools
ScrapeIn

Harvest employee email addresses from a specific company through LinkedIn.

urlgrab on offsec.tools
urlgrab

A golang utility to spider through a website searching for additional links.

crawley on offsec.tools
crawley

The unix-way web crawler.

skipfish on offsec.tools
skipfish

Active web application security reconnaissance tool.

CeWL on offsec.tools
CeWL

Custom Word List Generator.

ParamSpider on offsec.tools
ParamSpider

Mining parameters from dark corners of Web Archives.

gitpillage on offsec.tools
gitpillage

Extract data from a .git directory.

gau on offsec.tools
gau

Fetch known URLs from several sources.

hakrawler on offsec.tools
hakrawler

Simple, fast web crawler designed for discovery of endpoints and assets within a web application.