Is this crawler safe to use on production websites?

Yes. It features built-in rate limiting using a token bucket algorithm, configurable worker counts, and native respect for robots.txt rules to ensure server-friendly crawling.

How does it help with AI and RAG pipelines?

It includes a specialized Markdown export mode that strips away navigation, ads, and scripts, leaving clean content with YAML frontmatter that is optimized for LLM consumption.

What output formats are supported?

The crawler supports a wide range of formats including JSON, Markdown, CSV, plain text link lists, and an interactive HTML dashboard with graph visualizations.

How fast is the Claude Code Web Crawler?

Built in pure Rust, the crawler can process over 60 pages per second with near-instant startup times and a very low memory footprint of approximately 50MB.

Rust Web Crawler

Name: Rust Web Crawler
Author: leobrival

byleobrival

0•

Web Scraping & Data Collection

Extracts website content and structure at high speeds to generate LLM-ready Markdown, interactive site maps, and structured data for RAG pipelines.

The Web Crawler skill is a high-performance, Rust-powered tool designed to map and scrape websites with production-grade reliability and minimal resource overhead. It excels at transforming complex site structures into clean, LLM-optimized Markdown files and structured JSON data, making it an essential companion for developers building RAG pipelines, performing deep site audits, or migrating content. With built-in stealth features, automatic sitemap discovery, and strict robots.txt compliance, it provides a safe, scalable, and efficient way to gather web data without triggering bot protections or overloading target servers.

Key Features

01Intelligent sitemap discovery and automated robots.txt compliance checking.

02High-speed 60+ pages/sec throughput powered by an async Rust Tokio runtime.

03Advanced stealth mode with user-agent rotation and realistic request headers.

040 GitHub stars

05LLM-ready Markdown export featuring clean content filtering and YAML frontmatter.

06Interactive HTML reports with graph-based site structure and link visualizations.

Use Cases

01Building high-quality datasets for LLM training and RAG (Retrieval-Augmented Generation) applications.

02Automating large-scale content migrations by converting web pages into structured Markdown and JSON.

03Generating visual site maps and performing comprehensive link validation and structure analysis.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add leobrival/topographic-plugins-official web-crawler

For use in Claude.ai and ChatGPT

Download Skill