Popular GitHub Repositories for Webcrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

:sparkles: :dna: Turing ES - Enterprise Search, Semantic Navigation, Chatbot using Search Engine and Generative AI.

CodeBRT is an AI program generation plugin for VSCode. It helps you quickly generate code through AI, thus improving development efficiency.

Open Source Deep Research

An autonomous AI agent that crawls, summarizes, and publishes news articles.?

Predict the BitCoin's currency by using Altcoin currency and Web News.

RAG and Webcrawler in a single package

An ai agent that retrieves data from an islamic website [in Turkish language] -> processes the data -> utilizes an open ai NLP to answer the user

DevCrawler - An LLM Friendly Web Crawler & Data Scraper

RegRadar watches the worlds regulators so you dont have to.

该项目是基于网络爬虫，首先通过对百度搜索引擎对关键字进行搜索进而获取最实时性的消息，然后将获取到的消息创建为知识库，在调用deepseek模型时使用知识库中的内容让模型基于该内容进行回答

Together Open Deep Research