Popular GitHub Repositories for Deduplicate Data

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

? Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 80+ languages.

Deep Learning for humans

1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. ?Docker-friendly.?Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

LlamaIndex is the leading framework for building LLM-powered agents over your data.

Learn how to design, develop, deploy and iterate on production-grade ML applications.

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Build and share delightful machine learning apps, all in Python. ? Star to support our work!

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

AI's query engine - Platform for building AI that can learn and answer questions over large scale federated data.

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.

Collection of various algorithms in mathematics, machine learning, computer science and physics implemented in C++ for educational purposes.

Easily train a good VC model with voice data <= 10 mins!

? Industrial-strength Natural Language Processing (NLP) in Python

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.