Popular GitHub Repositories for Webcrawling

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

Crawl4DeepSeek = Crawl4AI + DeepSeek ? Smart, efficient, and built for deep web exploration! ??

The project uses Memory Based- RAG for healthcare queries, searching FAISS vector database for relevant answers. If no results are found, an AI fallback mechanism steps in. The AI Agent employs Selenium headless drivers, automation, web scraping, etc techniques to enhance search efficiency, ensuring accurate, real-time responses.