Does this skill handle JavaScript-heavy websites?

Yes, it includes Playwright templates specifically designed for full browser rendering and interaction with dynamic, JavaScript-heavy content.

Can I use this for RAG pipelines?

Absolutely. The skill includes scripts that automatically convert scraped content into clean, structured Markdown, which is ideal for RAG knowledge bases and LLM context.

How does it handle rate limiting?

It features built-in rate limiting utilities and templates that respect robots.txt, implement configurable delays, and handle 429 'Too Many Requests' errors gracefully.

What frameworks are supported?

It supports three major Python libraries: Playwright for browser automation, BeautifulSoup for lightweight static parsing, and Scrapy for complex, large-scale crawling.

Web Scraping & Content Extraction Tools

Name: Web Scraping & Content Extraction Tools
Author: vanman2024

byvanman2024

0•

Web Scraping & Data Collection

Automates documentation collection and structured data extraction using Playwright, BeautifulSoup, and Scrapy templates.

This skill provides a comprehensive suite of production-ready scraping tools designed for building RAG knowledge bases and collecting technical documentation. It features a decision matrix for choosing between Playwright for dynamic sites, BeautifulSoup for static parsing, and Scrapy for large-scale crawls. With built-in rate limiting, error handling, and markdown conversion, it ensures ethical and efficient data harvesting while respecting website resources and robots.txt protocols.

Key Features

01Multi-framework support for Playwright, BeautifulSoup, and Scrapy

02Automated markdown conversion for RAG-ready knowledge bases

030 GitHub stars

04Robust error handling with exponential backoff and request retries

05Built-in ethical scraping patterns including rate limiting and jitter

06Specialized scripts for technical documentation and article extraction

Use Cases

01Building and updating RAG knowledge bases from online technical documentation

02Aggregating blog posts or news articles for automated content analysis

03Monitoring competitor websites for content updates and new releases

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add vanman2024/ai-dev-marketplace web-scraping-tools

For use in Claude.ai and ChatGPT

Download Skill