Popular GitHub Repositories for Ingestion Pipeline

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Cross-platform, customizable ML solutions for live and streaming media.

?? Build multimodal AI applications with cloud-native stack

Turns Data and AI algorithms into production-ready web applications in no time.

? Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

File Parser optimised for LLM Ingestion with no loss ? Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.

Evidently is ??an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

An in-depth machine learning tutorial introducing readers to a whole machine learning pipeline from scratch.

A guideline for building practical production-level deep learning systems to be deployed in real world applications.

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，支持sso登录，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，标注平台，自动化标注，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式，deepseek训练推理

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

MedicalGPT: Training Your Own Medical GPT Model with ChatGPT Training Pipeline. 训练医疗大模型，实现了包括增量预训练(PT)、有监督微调(SFT)、RLHF、DPO、ORPO、GRPO。

Machine Learning Pipelines for Kubeflow

DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.

Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

? ????? about ????, ??????, and ?????? ??? for free by designing, training, and deploying a real-time financial advisor LLM system ~ ?????? ???? + ????? & ??????? ?????????