Popular GitHub Repositories for Webscraper

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

An AI assistant tool that integrates coding, writing, and reading functions. For better alternatives see https://monica.im/desktop

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

Financial Web Scraper & Sentiment Classifier

The spider crawls moneycontrol.com and economictimes.com to fetch news of input companies and also scores and classifies the companies to raise an early warning signal

The Web Metadata Extraction Toolkit is designed to streamline the process of extracting, cleaning, and analyzing metadata from websites. Utilizing advanced AI models and custom extraction strategies, this toolkit helps users efficiently gather data like titles, descriptions, and keywords, which are crucial for SEO and content strategy.

Applied Machine Learning and Natural Language Processing to build a Random Forest Classifier that filters malicious bots with 97% of accuracy.

A web scraping application which summarizes scraped information using AI technology

用于网页抓取的 ScrAPI 服务器。

Effortless Data Extraction, Powered by : Generative AI

AI News Scraper & Semantic Search: A Python application that scrapes news articles, uses GenAI to generate summaries and identify topics, and provides semantic search capabilities through vector embeddings

An intelligent web research assistant that combines Google Gemini, Google CSE API, BeautifulSoup-based scraping, TF-IDF ranking, and Chroma vector store to conduct deep web research and synthesize insights with proper citations.

A Python web scraper for BBC News using Selenium and BeautifulSoup4, powered by AI prompt engineering with ChatGPT. It extracts article data like URLs, headlines, and images, handles dynamic content, and outputs clean JSON. Includes Docker support for easy deployment.

(In-progress) Finetuning OpenAI's GPT-3.5-Turbo as a base model on open-source data about the Tampa Bay region to create a chatbot specializing in information on the area!

My n8n Automations during my internship at Crowdsource. My role is an Automation Specialist and is directly getting the task from the CEO.

A machine learning project that accurately predicts the price of any used car utilizing scraped car data from Craigslist. User will need to pull updated data using webscraper if requiring up-to-date data.

Simple webscraper that will utilize your local LLM to get a personalized result from scrapped website.

WebScraper que usa a API do Google Gemini para analisar e catalogar professores da UFABC conforme formação e área de interesse

Paginated Web Scraper – A large-scale, fault-tolerant Selenium scraper that extracts agent data from 5K+ paginated pages. Includes automatic backups, unique indexing, logging, and resume support.