Can I crawl an entire website or just single pages?

The skill supports deep crawling using breadth-first (BFS) or depth-first (DFS) strategies, allowing you to specify a maximum number of pages to process automatically.

Can Crawl4AI handle sites built with React or Vue?

Yes, it includes specialized configurations and JS wait conditions specifically designed to handle JavaScript-heavy Single Page Applications (SPAs) where content loads asynchronously.

How does the structured data extraction work?

You can perform extraction in two ways: using LLM-based natural language instructions for flexibility, or providing a JSON schema for faster, deterministic results.

Do I need to write Python code to use this skill?

No, this skill utilizes the crwl CLI, allowing you to perform complex scrapes and structured extractions directly through terminal commands or Claude instructions.

Crawl4AI Web Scraper

Name: Crawl4AI Web Scraper
Author: durandom

bydurandom

0•

Web Scraping & Data Collection

Extracts clean markdown and structured data from any website, including JavaScript-heavy single-page applications.

This skill empowers Claude to perform sophisticated web crawling and data extraction using the Crawl4AI engine. It excels at converting complex documentation sites into readable markdown, scraping JavaScript-rendered pages like React or Obsidian Publish, and extracting structured JSON data via LLM instructions or predefined schemas. With built-in support for deep crawling, batch processing, and anti-bot bypass, it is an essential tool for gathering high-quality web data for RAG, analysis, or content migration without needing to write custom Python scripts.

Key Features

01Structured data extraction using LLM-based instructions or JSON schemas

02Multi-page deep crawling with breadth-first and depth-first strategies

030 GitHub stars

04Robust handling of JavaScript-heavy SPAs using custom wait conditions

05Pre-configured settings for common platforms like Obsidian Publish and slow sites

06Advanced markdown extraction with boilerplate filtering (md-fit)

Use Cases

01Automating data collection from modern JavaScript-rendered web applications

02Converting online documentation into local markdown files for AI training or RAG

03Scraping e-commerce product data and pricing into structured JSON format

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add durandom/skills crawl4ai

For use in Claude.ai and ChatGPT

Download Skill