Popular GitHub Repositories for Wikipedia Scraper

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

Turn any webpage into structured data using LLMs

A community-driven way to read and chat with AI bots - powered by chatGPT.

Easily collect responses from ChatGPT scraper by providing a prompt along with valid Web Scraper API credentials.

Oxylabs AI Studio python SDK

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages.

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

Browser4: a lightning-fast, coroutine-safe browser for your AI.

What If Language Models Expertly Routed All Inference? WilmerAI allows prompts to be routed to specialized workflows based on the domain chosen by your LLM. Also allows chat Assistants to be powered by multiple LLMs working in tandem to generate a response. Compatible with Socg's Offline Wikipedia Article API.

Retrieve, Read and LinK: Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (ACL 2024)

Semantic Search on Wikipedia with Upstash Vector

A multithreaded ?? web crawler that recursively crawls a website and creates a ? markdown file for each page, designed for LLM RAG

ADAM - A Question Answering System. Inspired from IBM Watson

Monitor and download Manga automatically in a Docker-Container.

Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.

The GPT-based Universal Web Scraper MVP is a solution that leverages GPT models and web scraping libraries to generate scraper code based on user input and website analysis, simplifying the web scraping process.

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.