Popular GitHub Repositories for Document Pipeline

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Cross-platform, customizable ML solutions for live and streaming media.

PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/Docker/Zotero

?? Build multimodal AI applications with cloud-native stack

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

the AI-native open-source embedding database

Turns Data and AI algorithms into production-ready web applications in no time.

? Easy-to-use and powerful NLP and LLM library with ? Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including ?Text Classification, ? Neural Search, ? Question Answering, ?? Information Extraction, ? Document Intelligence, ? Sentiment Analysis etc.

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/

? Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022

Evidently is ??an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

A Unified Toolkit for Deep Learning Based Document Image Analysis

docTR (Document Text Recognition) - a seamless, high-performing & accessible library for OCR-related tasks powered by Deep Learning.