How do I ensure I don't crawl marketing or blog pages?

The Documentation Crawler provides specific URL filtering patterns (like /docs/ or /api/) and exclusion rules (like /blog/ or /pricing/) to ensure only technical content is captured.

Do I need to install any external tools to use this skill?

Yes, this skill utilizes the 'inform' CLI tool for HTML to Markdown conversion. The skill provides instructions for installation via npm, Bun, or a direct shell script.

Does it support downloading plain text or existing Markdown files?

Yes. The skill identifies when documentation is already provided in text-based formats (.md, .txt, .rst) and recommends using faster methods like curl or wget instead of a full crawl.

Can I use the crawled content to build my own Claude Code skills?

Absolutely. The skill includes a dedicated decision tree and workflow specifically for transforming crawled documentation into Personal, Project, or Plugin-level skills.

Can it handle sites that don't have a sitemap.xml?

Yes. The skill includes a direct crawl workflow that attempts to discover documentation pages by following links even when a sitemap is unavailable.

Documentation Crawler

Name: Documentation Crawler
Author: EpochTime-AI

byEpochTime-AI

0•

Web Scraping & Data Collection

Automates the process of crawling, extracting, and converting documentation websites into structured Markdown or Claude Code skills.

The Documentation Crawler skill provides a systematic workflow for developers to ingest external documentation into their local environment or Claude Code setup. By leveraging the inform tool, it handles complex tasks like sitemap discovery, URL filtering to exclude non-essential pages, and batch crawling with configurable rate-limiting. This skill is essential for developers looking to create custom AI skills from existing product docs or maintain local, searchable versions of technical manuals without manual copy-pasting, ensuring your AI assistant always has access to the most accurate, domain-specific information.

Key Features

01High-performance batch crawling with 'inform' tool integration

02Built-in repository cleanup and .gitignore configuration for crawl artifacts

03Automated sitemap discovery and URL filtering for targeted documentation extraction

04Seamless conversion of HTML web pages into clean, structured Markdown files

05Step-by-step guidance for creating Personal, Project, or Plugin skills from crawled data

060 GitHub stars

Use Cases

01Systematically extracting tutorials and guides from a developer portal to build a domain-specific RAG system

02Creating a custom Claude Code skill for a specific library by crawling its official API reference

03Converting a legacy HTML documentation site into a modern Markdown knowledge base for local reference

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add epochtime-ai/cc-plugins doc-crawler

For use in Claude.ai and ChatGPT

Download Skill