The Crawl4AI RAG MCP Server is an AI agent service integrating web crawler and RAG functions, supporting smart URL detection, recursive crawling, parallel processing, and vector search. It aims to provide powerful knowledge acquisition and retrieval capabilities for AI coding assistants.

The MediaCrawler MCP service upgrades the social media crawler into a standardized tool that can be directly called by AI assistants. It supports multi - platform data acquisition and has features such as externalized login, browser reuse, and structured output.

Deployment Guide for Web Crawler MCP Server

PodCrawlerMCP is an MCP service that discovers podcast content through web crawlers, helping AI assistants find podcast shows and episodes based on topics.

WebSearch-MCP is a service that implements the Model Context Protocol (MCP). It provides web search capabilities by integrating the WebSearch Crawler API and supports multiple AI clients to obtain web information in real-time.

An MCP service that connects the Claude desktop version with the local eGet web crawler, enabling the function of scraping web page content through the local API

A Python-based MCP web crawler project for extracting and saving website content as Markdown files, supporting batch processing and multi-threaded configuration.

A website security scanning tool based on the MCP protocol, integrating dirsearch directory scanning and firecrawl crawler technology, capable of automatically identifying the website technology stack and classifying vulnerability risk levels

An RSS crawler server based on the MCP protocol, used for crawling and managing RSS subscription content and integrating with LLM.

A development documentation server based on the MCP protocol, providing functions such as document crawling, local loading, precise search, and detail retrieval, to solve the document hallucination problem in AI development.

The Firecrawl MCP Server is a web crawler and data extraction service based on the Firecrawl API, providing functions such as web page scraping, content search, site crawling, and structured data extraction.

This project provides a set of tools for crawling website content and generating Markdown documents. It also implements semantic search for documents through the MCP server and supports integration with tools such as Cursor.

MCP Smart Crawler is a web content crawler based on Playwright, specifically used to extract metadata from Xiaohongshu posts and download media resources.

The Crawlab MCP server is a middleware connecting AI applications and the Crawlab crawler platform, enabling natural language interaction through a standardized protocol

Dafty MCP is an independently developed open - source project that interacts with Daft.ie through web crawlers, providing functions for searching and querying details of rental information in Ireland.

This project is a semantic search tool based on Elasticsearch for semantic retrieval of Search Labs blog posts, including crawler configuration, index mapping updates, and MCP server integration functions.

An MCP server that provides Google Search and web content viewing functions, with advanced anti - crawler evasion technology

mcp - server - webcrawl is an advanced web crawler data search and retrieval tool designed specifically for AI clients. It supports multiple crawler formats (such as WARC, wget, etc.), provides full - text search, Boolean logic queries, and resource type/status filtering functions. It can be seamlessly integrated with Claude Desktop, is installed via Python, and is suitable for tasks such as building website knowledge bases or conducting SEO/performance audits.

Crawl4AI is an intelligent web crawler server based on the MCP architecture. It combines the Claude AI model to provide website content analysis and processing functions, supports customizable crawling depth and AI processing tasks, and is suitable for various scenarios such as research, content creation, and data analysis.

A pure crawler - based web search MCP service based on Node.js, supporting Bing web page and news searches without an official API, providing high - performance crawling and health monitoring functions.