Popular GitHub Repositories for Inference Performance

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

A high-throughput and memory-efficient inference and serving engine for LLMs

Port of OpenAI's Whisper model in C/C++

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A generative speech model for daily dialogue.

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference

Cross-platform, customizable ML solutions for live and streaming media.

SoftVC VITS Singing Voice Conversion

Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

ncnn is a high-performance neural network inference framework optimized for the mobile platform

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

SGLang is a fast serving framework for large language models and vision language models.

Machine learning, in numpy

High-performance In-browser LLM Inference Engine