Popular GitHub Repositories for Data Engineering Pipeline

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

21 Lessons, Get Started Building with Generative AI ? https://microsoft.github.io/generative-ai-for-beginners/

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

Deep Learning for humans

? Guides, papers, lecture, notebooks and resources for prompt engineering

1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

? Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

LlamaIndex is the leading framework for building LLM-powered agents over your data.

Learn how to design, develop, deploy and iterate on production-grade ML applications.

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Build and share delightful machine learning apps, all in Python. ? Star to support our work!

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

AI's query engine - Platform for building AI that can learn and answer questions over large scale federated data.

Collection of various algorithms in mathematics, machine learning, computer science and physics implemented in C++ for educational purposes.

? Industrial-strength Natural Language Processing (NLP) in Python

Easily train a good VC model with voice data <= 10 mins!

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Cross-platform, customizable ML solutions for live and streaming media.