Popular GitHub Repositories for Rust Scraping

? ChatGPT Desktop Application (Mac, Windows and Linux)

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.

AI Code Completions

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI Tools & Agents, with access to OpenAI, Claude, Gemini, Ollama, Groq, and more.

一款专注于记录和写作的跨端 AI 笔记应用。

The world’s first real-time, distributed, cloud-edge collaborative multimodal AI Agent Framework that simultaneously supports C/C++/Go/Python/JS/TS

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Blazingly fast LLM inference.

The AI framework that adds the engineering to prompt engineering (Python/TS/Ruby/Java/C#/Rust/Go compatible)

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Vim和Neovim的免费、超高速Copilot替代方案

Free, ultrafast Copilot alternative for Vim and Neovim

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

面向应用开发者的开源响应式数据库

Noise supression using deep filtering