Popular GitHub Repositories for Document Ingestion

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.

PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/Docker/Zotero

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

the AI-native open-source embedding database

? Easy-to-use and powerful NLP and LLM library with ? Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including ?Text Classification, ? Neural Search, ? Question Answering, ?? Information Extraction, ? Document Intelligence, ? Sentiment Analysis etc.

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/

File Parser optimised for LLM Ingestion with no loss ? Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.

Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022

A Unified Toolkit for Deep Learning Based Document Image Analysis

docTR (Document Text Recognition) - a seamless, high-performing & accessible library for OCR-related tasks powered by Deep Learning.

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction

??AutoDev: The AI-powered coding wizard（AI 驱动编程助手）with multilingual support ?, auto code generation ??, and a helpful bug-slaying assistant ?! Customizable prompts ? and a magic Auto Dev/Testing/Document/Agent feature ? included! ?

An automated document analyzer for Paperless-ngx using OpenAI API, Ollama, Deepseek-r1, Azure and all OpenAI API compatible Services to automatically analyze and tag your documents.

Document Chatbot — multiple files. Powered by GPT / Embedding.

Top2Vec learns jointly embedded topic, document and word vectors.

Scalable Multi-modal RAG platform

A Repo For Document AI

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.