Popular GitHub Repositories for News Scraping

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

[?updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. ? online docs: https://ufund-me.github.io/Qbot ? :news: qbot-mini: https://github.com/Charmve/iQuant

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

A complete guide to start and improve in machine learning (ML), artificial intelligence (AI) in 2025 without ANY background in the field and stay up-to-date with the latest news and state-of-the-art techniques!

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

Single powerful tool to convert ANY documentation website into a Claude skill

A community-driven way to read and chat with AI bots - powered by chatGPT.

? 热门新闻聚合分析工具 - 自媒体微信公众号等新闻类文章辅助工具，支持今日头条、百度热搜、微博、抖音、知乎、B 站、财联社等11个中文热门平台，分析关键词频率，生成报告并支持飞书推送等。持续追踪热点变化。【欢迎 star ⭐支持开发者，后期考虑增加企业微信，钉钉，telegram等推送渠道，欢迎支持】【不定期更新功能】

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis

An AI-driven Hacker Podcast project that automatically fetches top Hacker News articles daily, generates summaries using AI, and converts them into podcast episodes

一个基于 AI 的 Hacker News 中文播客项目，每天自动抓取 Hacker News 热门文章，通过 AI 生成中文总结并转换为播客内容。

Fully functional Pokerbot that works on PartyPoker, PokerStars and GGPoker, scraping tables with Open-CV (adaptable via gui) or neural network and making decisions based on a genetic algorithm and montecarlo simulation for poker equity calculation. Binaries can be downloaded with this link:

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Oxylabs AI Studio python SDK

(Supports DeepSeek R1) An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models.

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama