Popular GitHub Repositories for Mixture

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Machine learning, in numpy

Optimizing inference proxy for LLMs

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

Mixture-of-Experts for Large Vision-Language Models

MoBA: Mixture of Block Attention for Long-Context LLMs

?? LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

Tutel MoE: An Optimized Mixture-of-Experts Implementation

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

An extension of XGBoost to probabilistic modelling

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

DanceNet -??Dance generator using Autoencoder, LSTM and Mixture Density Network. (Keras)

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

Tiled Diffusion, MultiDiffusion, Mixture of Diffusers, and optimized VAE

Implementation of "Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image"

Mixture of Diffusers for scene composition and high resolution image generation

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

A generic Mixture Density Networks (MDN) implementation for distribution and uncertainty estimation by using Keras (TensorFlow)