Popular GitHub Repositories for Experts

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets

Optimizing inference proxy for LLMs

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

Mixture-of-Experts for Large Vision-Language Models

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

?? LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

Tutel MoE: An Optimized Mixture-of-Experts Implementation

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

OpenAGI: When LLM Meets Domain Experts

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

This is the official repo for "PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization". PromptAgent is a novel automatic prompt optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts, i.e., expert-level prompts.

An open-source text summarization toolkit for non-experts. EMNLP'2021 Demo

MoH: Multi-Head Attention as Mixture-of-Head Attention

Offical Repo for "Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale"