LLM-Data-Pipeline

Scrapy, a fast high-level web crawling & scraping framework for Python.

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Elegant Scraper and Crawler Framework for Golang

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

A Chrome DevTools Protocol driver for web automation and scraping.

? Finding duplicate images made easy!

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

Headless Chrome .NET API

A list of AI agents and robots to block.

End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow