Popular GitHub Repositories for Inferentia

A high-throughput and memory-efficient inference and serving engine for LLMs

Large-scale LLM inference engine

A user-friendly Command-line/SDK tool that makes it quickly and easier to deploy open-source LLMs on AWS

This Guidance demonstrates how to deploy a machine learning inference architecture on Amazon Elastic Kubernetes Service (Amazon EKS). It addresses the basic implementation requirements as well as ways you can pack thousands of unique PyTorch deep learning (DL) models into a scalable architecture and evaluate performance

Collection of bet practices, reference architectures, examples, and utilities for foundation model development and deployment on AWS.

Scalable multimodal AI system combining FSDP, RLHF, and Inferentia optimization for customer insights generation.