Popular GitHub Repositories for Fast Inference

Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.

This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"

Demo code for CVPR2023 paper "Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers"