Does it automatically find sitemaps?

Yes, the crawler is designed to automatically discover and parse sitemap.xml files to ensure comprehensive coverage of all indexed pages.

Can the crawler handle large websites?

Yes, it uses a high-performance Go backend for concurrent crawling and includes checkpoint/resume functionality so you don't lose progress on large domains.

What output formats are supported?

The crawler generates a structured results.json for data analysis and a dark-themed, mobile-responsive HTML report for a visual summary of the crawl stats.

How does this crawler avoid getting blocked by servers?

The skill implements a token bucket rate-limiting algorithm and offers a 'gentle' profile that reduces concurrency and request frequency to respect server limits.

Website Crawler

Name: Website Crawler
Author: leobrival

byleobrival

0•

Web Scraping & Data Collection

Maps complex website structures and discovers URLs using a high-performance Go-powered engine with TypeScript integration.

The Website Crawler skill provides a high-performance engine designed to map entire website architectures, discover internal/external links, and generate comprehensive site reports. By combining a TypeScript/Bun frontend with a concurrent Go backend, it efficiently handles large-scale crawls while offering advanced features like automatic sitemap discovery, token-bucket rate limiting, and persistent checkpointing to resume progress on long-running tasks. It is ideal for developers and SEO professionals needing a structured overview of a site's content and connectivity.

Key Features

01Auto-save checkpointing every 30 seconds to resume interrupted crawls

02Dark-themed HTML report generation with visual site statistics

03Automatic sitemap.xml discovery and parsing for thorough site mapping

04Token-bucket rate limiting to prevent server overload and IP blocking

05High-performance Go worker pool for high-concurrency crawling

060 GitHub stars

Use Cases

01Generating comprehensive sitemaps and mapping complex site hierarchies for SEO

02Extracting page-level metadata including titles, status codes, and link relationships

03Auditing website coverage to identify broken links and orphaned pages

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add leobrival/serum-plugins-official website-crawler

For use in Claude.ai and ChatGPT

Download Skill