How are the crawled files organized?

Files are saved in a flat directory structure within your project's crawled_context folder, with filenames automatically derived from the original URLs.

Can I target specific parts of a website?

Yes, the --instruction parameter allows you to provide natural language guidance to the crawler, such as 'Focus on API reference pages only'.

Can I control how many pages are crawled?

Yes, you can customize the crawl using the --depth, --breadth, and --limit parameters to control the scope and prevent excessive API usage.

Does this skill require an external API key?

Yes, a Tavily API key is required. You can obtain one from tavily.com and add it to your Claude Code settings.json file.

Web Crawler & URL Extractor

Name: Web Crawler & URL Extractor
Author: tavily-ai

bytavily-ai

•

Web Scraping & Data Collection

Crawls websites using the Tavily API to convert web pages into local markdown files for offline analysis and documentation retrieval.

This skill empowers Claude to systematically crawl entire websites and transform web content into structured markdown files stored locally within your project. By leveraging the Tavily Crawl API, it handles complex site structures and allows for natural language guidance to focus on specific content—such as API endpoints or specific modules—while managing crawl depth, breadth, and page limits. It is an essential tool for developers needing to ingest documentation, build local knowledge bases, or perform deep analysis of web-based resources directly within their development environment without manual effort.

Key Features

01Configurable crawl depth, breadth, and total page limits

02Automatic conversion of web content to clean markdown format

03Source URL and timestamp metadata included in every file

04Flat directory structure storage for easy file management

05Natural language guidance to focus the crawler on specific topics

061 GitHub stars

Use Cases

01Archiving web tutorials and technical guides into local project directories

02Downloading entire framework documentation sets for offline reference

03Extracting knowledge bases to provide context for AI-driven code analysis

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add tavily-ai/tavily-plugins crawl-url

For use in Claude.ai and ChatGPT

Download Skill