Popular GitHub Repositories for Kv Cache Quantization

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

用超强记忆力打造你自己的第二大脑。它就像你的书签专属ChatGPT。使用Chrome扩展程序导入推文或保存网站和内容。

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Fast inference engine for Transformer models

Supercharge Your LLM with the Fastest KV Cache Layer

Vector (and Scalar) Quantization, in Pytorch

Sparsity-aware deep learning inference runtime for CPUs

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Scalable Multi-modal RAG platform

? Accelerate inference and training of ? Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)

Build, customize and control you own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape