Crawls and extracts website content into structured markdown files or context-optimized chunks for AI analysis.
The Tavily skill enables Claude to autonomously browse, map, and extract data from entire websites or specific documentation sub-paths. It transforms web content into clean markdown, offering advanced features like semantic instructions to filter for relevant information, depth control to prevent runaway crawls, and a specialized 'Map' mode for rapid URL discovery. Whether you need to ingest a full knowledge base for offline use or provide precise technical context to an LLM without hitting token limits, this skill streamlines the process of turning the web into actionable data.
Key Features
01Deep recursive crawling with configurable depth and page limits
02Semantic guidance using natural language instructions to focus on specific content
03Context-optimized chunking to prevent LLM context window overflow
04Path filtering using regex to include or exclude specific site sections
05Automated URL mapping for rapid site structure discovery
0619 GitHub stars
Use Cases
01Downloading entire documentation sites as local markdown for offline reference or RAG pipelines
02Performing deep research on technical topics by extracting only relevant snippets from multiple pages
03Mapping out website architectures to identify API endpoints or specific guide URLs
