Popular GitHub Repositories for Throughput

A high-throughput and memory-efficient inference and serving engine for LLMs

Running large language models on a single GPU for throughput-oriented scenarios.

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

High throughput synchronous and asynchronous reinforcement learning

A throughput-oriented high-performance serving framework for LLMs

Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning Framework on a GPU (JMLR 2022)

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

A low-latency & high-throughput serving engine for LLMs

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

PyTorch Library for Low-Latency, High-Throughput Graph Learning on GPUs.

LLM Benchmark for Throughput via Ollama (Local LLMs)

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

active learning for accelerated high-throughput virtual screening

A performant high-throughput CPU-based API for Meta's No Language Left Behind (NLLB) using CTranslate2, hosted on Hugging Face Spaces.

LLM Serving Performance Evaluation Harness

Qualcomm Cloud AI SDK (Platform and Apps) enable high performance deep learning inference on Qualcomm Cloud AI platforms delivering high throughput and low latency across Computer Vision, Object Detection, Natural Language Processing and Generative AI models.

A Throughput-Optimized Pipeline Parallel Inference System for Large Language Models

TwinGraph is a Python framework for distributed container orchestration using Kubernetes clusters, Docker Compose/Swarm or cloud resources on AWS (AWS Lambda, AWS Batch, Amazon EKS). Applications include high-throughput simulations, simulation-driven optimization, Digital Twins and machine learning.

Official code for "Throughput-Optimal Topology Design for Cross-Silo Federated Learning" (NeurIPS'20)

mantis-ml: Stochastic semi-supervised learning to prioritise genes from high throughput genomic screens