Popular GitHub Repositories for Cache Aside

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

Supercharge Your LLM with the Fastest KV Cache Layer

Scalable Multi-modal RAG platform

Unified KV Cache Compression Methods for Auto-Regressive Models

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

LLM KV cache compression made easy

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

High performance distributed cache system. Built by Rust.

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

?A curated list of Awesome Diffusion Inference Papers with Codes: Sampling, Cache, Quantization, Parallelism, etc.?

Redis Vector Library (RedisVL) -- the AI-native Python client for Redis.

[ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference