Popular GitHub Repositories for News Crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

[?updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. ? online docs: https://ufund-me.github.io/Qbot ? :news: qbot-mini: https://github.com/Charmve/iQuant

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

A complete guide to start and improve in machine learning (ML), artificial intelligence (AI) in 2025 without ANY background in the field and stay up-to-date with the latest news and state-of-the-art techniques!

A community-driven way to read and chat with AI bots - powered by chatGPT.

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis

一个基于 AI 的 Hacker News 中文播客项目，每天自动抓取 Hacker News 热门文章，通过 AI 生成中文总结并转换为播客内容。

Flexible Node.js AI-assisted crawler library

? 热门新闻聚合分析工具 - 自媒体微信公众号等新闻类文章辅助工具，支持今日头条、百度热搜、微博、抖音、知乎、B 站、财联社等11个中文热门平台，分析关键词频率，生成报告并支持飞书推送等。持续追踪热点变化。【欢迎 star ⭐支持开发者，后期考虑增加企业微信，钉钉，telegram等推送渠道，欢迎支持】【不定期更新功能】

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Chat with Hacker News using natural language. Built with OpenAI Functions and Vercel AI SDK.

Chinese NewsTitle Generation Project by GPT2.带有超级详细注释的中文GPT2新闻标题生成项目。

A scalable, mature and versatile web crawler based on Apache Storm

Code for Defending Against Neural Fake News, https://rowanzellers.com/grover/

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

A complete guide to start and improve your LLM skills in 2025 with little background in the field and stay up-to-date with the latest news and state-of-the-art techniques!

Use NLP to predict stock price movement associated with news