Popular GitHub Repositories for Crawling Sites

A lightning-fast search engine API bringing AI-powered hybrid search to your sites and applications.

?免费的 ChatGPT 镜像网站列表，持续更新。List of free ChatGPT mirror sites, continuously updated.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

DeerFlow is a community-driven Deep Research framework, combining language models with tools like web search, crawling, and Python execution, while contributing back to the open-source community.

A list of AI agents and robots to block.

Base on YOLOv5 Head Person Helmet Detection on Construction Sites，基于目标检测工地安全帽和禁入危险区域识别系统，??附 YOLOv5 训练自己的数据集超详细教程??2021.3新增可视化界面❗❗

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Lightweight library for scraping web-sites with LLMs

This repository contains a list of free to use AI APIs and sites. We are NOT endorsing ANY of the listed services!

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal for large enterprises with recurring data extraction needs.

Static web application for generative artists to explore creative concepts via mappable hashpair control sliders, automate image generation and capture features information to CSV in order to analyze probability of outcomes.

one click to open multi AI sites ｜ 一键打开多个 AI 站点，查看 AI 结果

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web.

AI Side Panel Extension: Enhance your productivity with this browser extension that adds a convenient side panel for quick access to popular AI sites like Gemini, ChatGPT, Copilot, Claude, and Deepseek. Easily toggle the panel using the extension icon or ALT + Q shortcut. Perfect for seamless integration of AI tools into your browsing experience.

Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.

Web scraping API for building AI applications.

An agent with human in the loop that can search the web for information while bypassing bot detection for private sites.