WebPeel is a powerful web fetching tool designed specifically for AI agents, capable of converting any web page into clean, AI-ready markdown. It features smart escalation, automatically progressing from fast HTTP fetches to browser rendering, and even a stealth mode to bypass bot detection when necessary. With support for both Node.js and Python SDKs, a global CLI, and a self-hostable Docker setup, WebPeel offers robust web crawling, mapping, and searching capabilities. It also integrates seamlessly with AI tools via its MCP server, providing a suite of tools for autonomous agents to perform web research using your own LLM key.
Key Features
010 GitHub stars
02Smart Escalation (HTTP → Browser → Stealth)
03Stealth Mode for bypassing bot detection
04Web Crawling and Mapping with sitemap and robots.txt compliance
05Autonomous Agent with Bring Your Own Key (BYOK) LLM support
06Comprehensive MCP Server integration with 9 tools for AI agents
