Web Scraping & Data Collection Agent Skills

Twitter/X Media Downloader

Downloads high-quality images and videos from Twitter/X using automated gallery-dl workflows.

Google Places Search

Queries the Google Places API to search for locations, retrieve place details, and fetch reviews directly within the terminal.

Google Places Search

Integrates Google Places API into your workflow for searching locations, retrieving business details, and analyzing reviews.

RFP Data Ingestion Engine

Automates the collection, normalization, and deduplication of Request for Proposal (RFP) opportunities from government and private data sources.

Article Extractor

Extracts clean, readable text from web articles and blog posts by removing ads, navigation, and clutter.

Wget URL Reader

Fetches and downloads content from any URL using the powerful wget command-line utility.

Brave Search Integration

Performs headless web searches and extracts readable markdown content using the Brave Search API without requiring a browser.

Blockchain Data Pipeline Validation

Validates blockchain data collection pipelines using a systematic 5-step empirical workflow to ensure data integrity and storage efficiency.

Exa Semantic Web Search

Empowers Claude with semantic, neural search capabilities and specialized web filtering using the Exa API.

YouTube Video to Audio

Downloads and converts YouTube videos into high-quality audio files using yt-dlp and ffmpeg.

bioRxiv Preprint Search

Accesses and retrieves research papers from the bioRxiv preprint server for literature reviews and trend analysis.

Ebook Text Extractor

Extracts clean, plain text from EPUB, MOBI, and PDF files for analysis and data processing.

China Macro News Aggregator

Aggregates and summarizes real-time China macro-economic news from premium financial sources into professional magazine-style reports.

Smart Link Extractor

Extracts core resources from social media and technical blogs to generate structured Markdown archives with intelligent categorization.

WASDE Data Ingestor

Automates the extraction and parsing of monthly USDA WASDE reports into standardized datasets for agricultural market analysis.

Scraper Code Generator

Generates high-performance, robust Python code for scraping and parsing structured API documentation from HTML.

Doc Site Analysis

Analyzes API documentation structures to identify data extraction patterns for automated scraper generation.

WeChat Article Fetcher & Research Analyst

Fetches and ranks WeChat articles based on research interests with seamless Obsidian integration.

Valyu Search & Research Toolkit

Empowers Claude with multi-domain search, AI-driven answers, content extraction, and comprehensive deep research reporting.

Scanner Excel Extraction

Extracts, transforms, and structures data from complex Excel files into JSON or CSV formats.

Google Places CLI

Simplifies querying the Google Places API for search, location details, and business reviews directly from the terminal.

Scanner PDF Analysis

Analyzes PDF documents to extract structured data, including tables, section headers, and metadata, while providing automated summaries.

Blog & RSS Monitor

Monitors and manages updates from blogs and RSS/Atom feeds directly through the CLI.

Web Connectivity and Search

Searches the internet and converts live webpage content into markdown for real-time information retrieval and analysis.

Playwright Web Scraper

Extracts structured data from complex websites using a robust, three-phase Playwright automation workflow.

Job Description Parser

Extracts structured requirements and metadata from job descriptions to facilitate automated candidate matching and recruitment analysis.

Add Awesome Tool

Analyzes AI tool URLs to extract metadata and automatically categorizes and adds them to the awesome-ai-tools repository.

MCP Opportunity Pipeline

Automates the end-to-end lifecycle of discovering, validating, building, and publishing Model Context Protocol (MCP) servers and automation tools.

Web Search

Retrieves real-time information, news, images, and videos from the web using DuckDuckGo to provide up-to-date data and resources.

Research Reporter

Automates information gathering from web searches and authoritative sources to generate and save structured research reports.