Popular GitHub Repositories for Aigc Serving

A high-throughput and memory-efficient inference and serving engine for LLMs

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

SGLang is a fast serving framework for large language models and vision language models.

IM Chat ChatGPT

Awesome pre-trained models toolkit based on PaddlePaddle. (400+ models including Image, Text, Audio, Video and Cross-Modal with Easy Inference & Serving)

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in performant way.

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 15+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

LangChain 的中文入门教程

High-speed Large Language Model Serving for Local Deployment

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

? Yao is an all-in-one application engine that enables developers to create web apps, REST APIs, business applications, and more, with AI as a development partner.

OpenMMLab Multimodal Advanced, Generative, and Intelligent Creation Toolbox. Unlock the magic ?: Generative-AI (AIGC), easy-to-use APIs, awsome model zoo, diffusion models, for text-to-image generation, image/video restoration/enhancement, etc.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

? AIGC 提示词可视化编辑器 | OPS | Open Prompt Studio

A flexible, high-performance serving system for machine learning models

Multilingual Voice Understanding Model

AI + Data, online. https://vespa.ai