What happens if a site doesn't have an llms.txt file?

The skill includes a recovery protocol that detects missing files and suggests falling back to traditional documentation scraping or manual URL analysis.

What is the llms.txt standard?

The llms.txt standard is a convention for websites to provide Markdown-based documentation specifically optimized for Large Language Models in a predictable directory structure.

Why is this better than traditional web scraping?

It is significantly faster and more token-efficient because it retrieves clean Markdown content without HTML boilerplate, scripts, or navigational noise.

What file variants does this skill support?

It searches for llms-full.txt (complete docs), llms.txt (standard), and llms-small.txt (condensed) to match different context window requirements.

Can this skill be used for private documentation?

Yes, as long as the Claude Code environment has network access to the target URL, it can fetch and process any accessible llms.txt file.

llms.txt Support

Name: llms.txt Support
Author: jmagly

byjmagly

•

Web Scraping & Data Collection

Detects and ingests LLM-optimized documentation via the llms.txt standard to accelerate context gathering for autonomous agents.

The llms.txt Support skill streamlines documentation ingestion by prioritizing LLM-friendly formats over traditional, noisy web scraping. By detecting and processing standardized llms.txt, llms-full.txt, and llms-small.txt files, it allows Claude to quickly understand libraries and APIs using high-density, pre-formatted context. This skill includes robust grounding checkpoints and recovery protocols to mitigate common LLM failure modes, ensuring your agent always works with verified, high-quality reference material while minimizing token usage and latency.

Key Features

0167 GitHub stars

02Hybrid integration capabilities with traditional doc-scrapers for gap filling

03Automatic content validation to prevent ingestion of 404 error pages or redirects

04Smart variant selection based on context window limits and coverage needs

05Hierarchical detection of llms-full.txt, llms.txt, and llms-small.txt variants

06Multi-location probing including root, /docs, and .well-known directories

Use Cases

01Rapidly onboarding an AI agent to a new framework or library with existing llms.txt files

02Reducing token costs and runtime by fetching condensed documentation summaries

03Automating documentation discovery for large-scale multi-agent coding tasks

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jmagly/ai-writing-guide llms-txt-support

For use in Claude.ai and ChatGPT

Download Skill