Popular GitHub Repositories for Cache Aside

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

Supercharge Your LLM with the Fastest KV Cache Layer

Scalable Multi-modal RAG platform

Unified KV Cache Compression Methods for Auto-Regressive Models

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

LLM KV cache compression made easy

kvcached: Elastic KV cache for dynamic GPU sharing and efficient multi-LLM inference.

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

High performance distributed cache system. Built by Rust.

[NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

?A curated list of Awesome Diffusion Inference Papers with Codes: Sampling, Cache, Quantization, Parallelism, etc.?

Redis Vector Library (RedisVL) -- the AI-native Python client for Redis.

[ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache