Q-LLM

Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.

Unified KV Cache Compression Methods for Auto-Regressive Models

SpargeAttention: A training-free sparse attention that can accelerate any model inference.

Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.

[NeurIPS 2024] AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising

Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)

?? A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.

[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection

xKV: Cross-Layer SVD for KV-Cache Compression

Demo code for CVPR2023 paper "Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers"