Popular GitHub Repositories for Parallelism

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

? Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

?A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, Flash-Attention, Paged-Attention, Parallelism, etc. ??

Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications (MIT 18.337J/6.338J)

Must read research papers and links to tools and datasets that are related to using machine learning for compilers and systems optimisation

A GPipe implementation in PyTorch

[CVPR 2024 Highlight] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

Long-RL: Scaling RL to Long Sequences

Pax is a Jax-based machine learning framework for training large scale models. Pax allows for advanced and fully configurable experimentation and parallelization, and has demonstrated industry leading model flop utilization rates.

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

Data-driven model reduction library with an emphasis on large scale parallelism and linear subspace methods

[MLSys 2021] IOS: Inter-Operator Scheduler for CNN Acceleration

Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)

multi-gpu pre-training in one machine for BERT without horovod (Data Parallelism)