Popular GitHub Repositories for Crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

A community-driven way to read and chat with AI bots - powered by chatGPT.

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

Flexible Node.js AI-assisted crawler library

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

A scalable, mature and versatile web crawler based on Apache Storm

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情感分析; 5.主题聚类

:newspaper: Let ChatGPT Summarize Hacker News for You

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Medical Graph for Neo4j

Opensource Korean chatbot framework

A multithreaded ?? web crawler that recursively crawls a website and creates a ? markdown file for each page, designed for LLM RAG

Selenium Open Source Search Engine & crawler

data resource untuk NLP bahasa indonesia

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

利用文本挖掘技术进行新闻热点关注问题分析

Automatically constructing corpus for automatic speech recognition from YouTube videos