Popular GitHub Repositories for Recursive Crawling

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

DeerFlow is a community-driven Deep Research framework, combining language models with tools like web search, crawling, and Python execution, while contributing back to the open-source community.

A list of AI agents and robots to block.

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Tree LSTM implementation in PyTorch

Tensorflow implementation of pixel-recursive-super-resolution(Google Brain paper: https://arxiv.org/abs/1702.00783)

[ICCV 2019] Recursive Cascaded Networks for Unsupervised Medical Image Registration

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

Local LLM Powered Recursive Search & Smart Knowledge Explorer

Pytorch code for our ICLR 2017 paper "Layered-Recursive GAN for image generation"

Implementation of my paper "Real-time Document Localization in Natural Images by Recursive Application of a CNN."

Official code for "Recursive Video Lane Detection", ICCV 2023

ReDel is a toolkit for researchers and developers to build, iterate on, and analyze recursive multi-agent systems. (EMNLP 2024 Demo)

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal for large enterprises with recurring data extraction needs.

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web.

Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.

An engine for generating recursive documentation and using it to work on a project with ChatGPT

[CVPRW 2020] Focus Longer to See Better:Recursively Refined Attention for Fine-Grained Image Classification