Popular GitHub Repositories for Zio Cache

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

Scalable Multi-modal RAG platform

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

Unified KV Cache Compression Methods for Auto-Regressive Models

A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.

LLM KV cache compression made easy

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

Redis Vector Library (RedisVL) -- the AI-native Python client for Redis.

[ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

Vector Storage is a vector database that enables semantic similarity searches on text documents in the browser's local storage. It uses OpenAI embeddings to convert documents into vectors and allows searching for similar documents based on cosine similarity.

A client side vector search library that can embed, store, search, and cache vectors. Works on the browser and node. It outperforms OpenAI's text-embedding-ada-002 and is way faster than Pinecone and other VectorDBs.

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference