Popular GitHub Repositories for Dnn Serving

A high-throughput and memory-efficient inference and serving engine for LLMs

AiLearning：数据分析+机器学习实战+线性代数+PyTorch+NLTK+TF2

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

SGLang is a fast serving framework for large language models and vision language models.

Open standard for machine learning interoperability

Awesome pre-trained models toolkit based on PaddlePaddle. (400+ models including Image, Text, Audio, Video and Cross-Modal with Easy Inference & Serving)

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in performant way.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 15+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Experience, Learn and Code the latest breakthrough innovations with Microsoft AI

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Go package for computer vision using OpenCV 4 and beyond. Includes support for DNN, CUDA, OpenCV Contrib, and OpenVINO.

AI + Data, online. https://vespa.ai

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

A flexible, high-performance serving system for machine learning models

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

A Streamlit-powered Chat Assistant utilizing Ollama's DeepSeek-R1 (1.5B) model for intelligent, context-aware conversations. Designed with an interactive chat interface, adjustable model parameters. It leverages Ollama for local model serving and Streamlit for a responsive UI for text-based interactions.

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.

Standardized Serverless ML Inference Platform on Kubernetes