How does this skill differ from a standard web scraper?

This skill is specifically optimized for AI context. It doesn't just pull HTML; it organizes data into categorized markdown files following specific patterns that make the information more 'digestible' for Claude and other LLMs.

Do I need to be an expert in CSS selectors to use this?

While knowledge of selectors helps for complex sites, the skill provides configuration templates and guidance for identifying the main content, titles, and code blocks of most modern documentation sites.

How do I ensure I'm not violating a site's terms of service?

The skill includes a 'Grounding Checkpoint' that requires verification of robots.txt and supports configurable rate limiting to ensure respectful scraping practices.

Can I resume a scrape if it gets interrupted?

Yes, the skill includes a recovery protocol and checkpoint support. It saves state to a local directory, allowing you to resume from where you left off without re-scraping the entire site.

Documentation Scraper

Name: Documentation Scraper
Author: jmagly

byjmagly

•

Web Scraping & Data Collection

Transforms documentation websites into structured, categorized reference files optimized for AI context and offline archives.

The Documentation Scraper skill provides a robust framework for converting web-based documentation into high-quality, searchable reference data. It automates the extraction process using configurable CSS selectors and URL patterns, mapping site content into logical categories like API references, guides, and tutorials. Designed specifically for agentic coding workflows, it includes built-in mitigations for common LLM failure modes, such as grounding checks via robots.txt verification and uncertainty escalation when selectors are ambiguous, ensuring reliable output for building Claude skills or local knowledge bases.

Key Features

01Configurable CSS selectors for precise extraction of main content, titles, and code blocks

02Automatic content categorization using keyword mapping and directory structures

03Pattern-based URL filtering to include or exclude specific documentation paths

0467 GitHub stars

05Resumable progress tracking via a dedicated checkpoint system for large-scale scrapes

06Agentic safety protocols including rate limiting, robots.txt compliance, and grounding checks

Use Cases

01Creating offline, structured references for complex API documentation to reduce token usage

02Standardizing documentation from multiple web sources into a unified markdown format for AI training

03Building custom Claude Code skills by scraping official library or framework documentation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jmagly/ai-writing-guide doc-scraper

For use in Claude.ai and ChatGPT

Download Skill