Popular GitHub Repositories for Optimizing Pipelines

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. ?Docker-friendly.?Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.

Turns Data and AI algorithms into production-ready web applications in no time.

Unified framework for building enterprise RAG pipelines with small, specialized models

Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.

TensorZero creates a feedback loop for optimizing LLM applications — turning production data into smarter, faster, and cheaper models.

A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.

OpenVINO? is an open source toolkit for optimizing and deploying AI inference

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

All-in-One Development Tool based on PaddlePaddle（飞桨低代码开发工具）

ZenML ?: The bridge between ML and Ops. https://zenml.io.

?? CML - Continuous Machine Learning | CI/CD for ML

Machine Learning Pipelines for Kubeflow

MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

A system for agentic LLM-powered data processing and ETL

Optimizing inference proxy for LLMs

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.