Popular GitHub Repositories for Crawlers

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

A list of AI agents and robots to block.

? A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

Python course for 2nd year NLP students at NRU HSE, 2018-2019

Python course for 2nd year NLP students at NRU HSE, 2017-2018

一款强大的大模型微调数据集生成和管理工具。

Вседържец/Vsy - The AGI Infrastructure of "The Sacred Computer" AGI Institute : Custom Intelligent Selective Internet Archiving and Exploration/Crawling; Information Retrieval, Media Monitoring, Search Engine, Smart DB, Data Preservation, Knowledge Extraction,Datasets creation,AI Generative models building and testing,Experiments etc.