Popular GitHub Repositories for Inference Server

? Light and Fast AI Assistant. Support: Web | iOS | MacOS | Android | Linux | Windows

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

A high-throughput and memory-efficient inference and serving engine for LLMs

下一代 Node.js 和 TypeScript 对象关系映射 (ORM) 工具，支持 PostgreSQL、MySQL、MariaDB、SQL Server、SQLite、MongoDB 和 CockroachDB 数据库。

Port of OpenAI's Whisper model in C/C++

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A generative speech model for daily dialogue.

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference

Cross-platform, customizable ML solutions for live and streaming media.

? 1Panel offers an intuitive web interface for managing websites, files, containers, databases and LLMs within a Linux server.

SoftVC VITS Singing Voice Conversion

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

???AI-driven database tool and SQL client, The hottest GUI client, supporting MySQL, Oracle, PostgreSQL, DB2, SQL Server, DB2, SQLite, H2, ClickHouse, and more.

Context7 MCP 服务器——关于大型语言模型 (LLM) 和 AI 代码编辑器的最新文档

ncnn is a high-performance neural network inference framework optimized for the mobile platform

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）