Popular GitHub Repositories for Sparse Moe

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

SGLang is a fast serving framework for large language models and vision language models.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in performant way.

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors

Mixture-of-Experts for Large Vision-Language Models

MoBA: Mixture of Block Attention for Long-Context LLMs

? Gracefully face hCaptcha challenge with MoE(ONNX) embedded solution.

A package for the sparse identification of nonlinear dynamical systems from data

[MICRO'23, MLSys'22] TorchSparse: Efficient Training and Inference Framework for Sparse Convolution on GPUs.

[ICLR'23 Spotlight?] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

A super fast Graph Database uses GraphBLAS under the hood for its sparse adjacency matrix graph representation. Our goal is to provide the best Knowledge Graph for LLM (GraphRAG).

PyTorch Extension Library of Optimized Autograd Sparse Matrix Operations

Programmable CUDA/C++ GPU Graph Analytics

?? LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

Library for specialized dense and sparse matrix operations, and deep learning primitives.

SPLADE: sparse neural search (SIGIR21, SIGIR22)

Tutel MoE: An Optimized Mixture-of-Experts Implementation

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (CVPR 2023)

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI