How does this skill handle anti-bot measures?

The skill utilizes Playwright with stealth-mode plugins and rotates browser user agents to mimic human behavior, helping to avoid detection by common scraping guards.

What is 'Poison Pill' detection?

It is a mechanism to identify when a scraper has been redirected to a paywall, CAPTCHA, or login screen, ensuring the system doesn't treat 'junk' content as valid data.

What is a scraping cascade architecture?

It is a strategy that attempts fast, lightweight extraction methods first (like Trafilatura) and only falls back to resource-heavy methods (like Playwright) if the simpler ones fail or return insufficient data.

Can I use these scraping patterns in Jupyter Notebooks?

Yes, the skill specifically includes an asynchronous Playwright implementation designed to work within Jupyter's existing event loop, preventing common 'nested event loop' errors.

Advanced Web Scraping & Extraction

Name: Advanced Web Scraping & Extraction
Author: jamditis

byjamditis

•

Web Scraping & Data Collection

Extracts web content using a multi-tiered cascade architecture that bypasses anti-bot measures and detects paywalls.

This skill provides a robust framework for ethical and reliable web scraping, specifically designed for journalism, research, and data collection. It implements a sophisticated 'scraping cascade' that automatically falls back from lightweight tools like Trafilatura to heavy-duty headless browsers like Playwright with stealth mode. Beyond simple extraction, it includes patterns for reverse-engineering undocumented APIs, detecting 'poison pills' such as paywalls or CAPTCHAs, and handling specific media platforms like YouTube and Instagram. It is optimized for both standard Python scripts and asynchronous Jupyter Notebook environments.

Key Features

0141 GitHub stars

02Methodologies for discovering and reverse-engineering undocumented APIs

03Anti-bot bypass with Playwright stealth mode and rotating user agents

04Automated 'Poison Pill' detection for paywalls, CAPTCHAs, and rate limits

05Multi-tiered scraping cascade (Trafilatura -> Requests -> Playwright)

06Async Playwright support optimized for Jupyter Notebook (.ipynb) environments

Use Cases

01Extracting clean article text from news sites while bypassing paywall markers and truncated content.

02Automating large-scale data collection from social media and video platforms using established library patterns.

03Reverse-engineering search or autocomplete APIs to gather structured data without complex HTML parsing.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jamditis/claude-skills-journalism web-scraping

For use in Claude.ai and ChatGPT

Download Skill