Popular GitHub Repositories for Lead Scraping

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

Single powerful tool to convert ANY documentation website into a Claude skill

? 热门新闻聚合分析工具 - 自媒体微信公众号等新闻类文章辅助工具，支持今日头条、百度热搜、微博、抖音、知乎、B 站、财联社等11个中文热门平台，分析关键词频率，生成报告并支持飞书推送等。持续追踪热点变化。【欢迎 star ⭐支持开发者，后期考虑增加企业微信，钉钉，telegram等推送渠道，欢迎支持】【不定期更新功能】

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Fully functional Pokerbot that works on PartyPoker, PokerStars and GGPoker, scraping tables with Open-CV (adaptable via gui) or neural network and making decisions based on a genetic algorithm and montecarlo simulation for poker equity calculation. Binaries can be downloaded with this link:

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Oxylabs AI Studio python SDK

(Supports DeepSeek R1) An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models.

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

一个功能强大的模型上下文协议（MCP）服务器，提供一体化的公共网络访问解决方案。

? Experimental library for scraping websites using OpenAI's GPT API.

Lightweight library for scraping web-sites with LLMs

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

Extract any website’s design system into design tokens in a few seconds: logo, colors, typography, borders, and more. One command.

Browser4: a lightning-fast, coroutine-safe browser for your AI.

? AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph.