Popular GitHub Repositories for Data Engineering P

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

21 Lessons, Get Started Building with Generative AI ? https://microsoft.github.io/generative-ai-for-beginners/

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

? Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

? Guides, papers, lecture, notebooks and resources for prompt engineering

Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 80+ languages.

Deep Learning for humans

? Toolkit to help you get started with Spec-Driven Development

1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. ?Docker-friendly.?Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

LlamaIndex is the leading framework for building LLM-powered agents over your data.

Learn how to design, develop, deploy and iterate on production-grade ML applications.

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Build and share delightful machine learning apps, all in Python. ? Star to support our work!

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference

AI's query engine - Platform for building AI that can learn and answer questions over large scale federated data.

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.