Can this skill handle websites that use JavaScript?

Yes, it includes specific patterns for using Playwright and other browser automation tools to render dynamic content before extracting the required data.

Is the data extraction logic robust against site changes?

The skill teaches the use of multiple fallback selectors (CSS and XPath) and data validation techniques to ensure extraction remains functional even if a website's layout undergoes minor changes.

How does this skill help with legal compliance?

It provides standardized code patterns to check robots.txt files, respect Terms of Service, and prioritize official APIs to ensure your scraping activities remain within legal and ethical boundaries.

What techniques are used to avoid being blocked?

The skill implements several strategies including rotating User-Agent strings, managing session cookies, using proxy rotation, and respecting HTTP 429 'Retry-After' headers with exponential backoff.

Ethical Web Scraping & Data Extraction

Name: Ethical Web Scraping & Data Extraction
Author: cyperx84

bycyperx84

•

Web Scraping & Data Collection

Implements ethical, resilient, and legally compliant web scraping strategies to extract high-quality data while avoiding bot detection.

This skill provides a comprehensive framework for building production-ready web scrapers within the Claude Code environment. It emphasizes a 'polite-first' approach, guiding developers through legal compliance checks like robots.txt analysis and rate-limiting implementation via exponential backoff. Beyond ethics, the skill offers advanced technical patterns for bypassing common obstacles, including User-Agent rotation, session management, and JavaScript rendering for dynamic content. Whether you are using BeautifulSoup for static HTML or Playwright for complex web apps, this skill ensures your data extraction pipelines are robust, respectful, and reliable.

Key Features

01Legal and ethical compliance frameworks including robots.txt parsing

02Multi-layered data extraction strategies using CSS, XPath, and Regex

03Anti-bot evasion techniques using User-Agent and proxy rotation

04Sophisticated rate limiting and polite scraping implementation patterns

05Dynamic content extraction via JavaScript rendering and Playwright

062 GitHub stars

Use Cases

01Automating market research and competitor price monitoring

02Building data aggregators that require robust handling of diverse web structures

03Gathering large-scale datasets for academic research or ML training

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add cyperx84/claude-code-plugin-examples scraping-best-practices

For use in Claude.ai and ChatGPT

Download Skill