Provides a modular web crawler and a REST API for managing and storing webpage data, integrated with an MCP server.
Sponsored
This project serves as a comprehensive learning resource, featuring a robust web crawler designed for modularity and efficient task queuing. It's complemented by a powerful REST API, enabling full CRUD operations for webpage data, and an MCP server, facilitating seamless integration with external LLMs and AI agents. With automatic Swagger documentation, TypeORM/PostgreSQL integration, and comprehensive unit test coverage, it offers a solid foundation for developing intelligent data collection and management systems.
Key Features
01Modular web crawler with depth control and task queue
020 GitHub stars
03PostgreSQL integration via TypeORM
04MCP server for integration with external LLMs/agents via HTTP
05REST API for managing and storing webpage data (CRUD)
06Automatic API documentation and validation via Swagger and class-validator
Use Cases
01Automating web content collection and indexing for AI agents or LLMs
02Developing custom integrations for agents to access and process web information
03Persisting and managing crawled webpage data in a structured database
