Popular GitHub Repositories for Petabyte Scale

Making large AI models cheaper, faster and more accessible

AI's query engine - Platform for building AI that can learn and answer questions over large scale federated data.

Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

This repository delivers end-to-end, code-first tutorials covering every layer of production-grade GenAI agents, guiding you from spark to scale with proven patterns and reusable blueprints for real-world launches.

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.

Ongoing research training transformer models at scale

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

[NeurIPS 2024 Best Paper][GPT beats diffusion?] [scaling laws in visual generation?] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

A Flexible and Powerful Parameter Server for large-scale machine learning

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

A large-scale 7B pretraining language model developed by BaiChuan-Inc.

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.

? Super-scale your images and run experiments with Residual Dense and Adversarial Networks.

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

Lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.

? ? ? ? GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统

A high-performance distributed training framework for Reinforcement Learning