Popular GitHub Repositories for Attention Is All You Need

A PyTorch implementation of the Transformer model in "Attention is All You Need".

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.

list of efficient attention modules

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Open-Source Toolkit for End-to-End Korean Automatic Speech Recognition leveraging PyTorch and Hydra.

A Benchmark of Text Classification in PyTorch

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images

Attention Is All You Need | a PyTorch Tutorial to Transformers

Implementation of "PaLM-E: An Embodied Multimodal Language Model"

Original transformer paper: Implementation of Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017.

Pytorch implementation of the models RT-1-X and RT-2-X from the paper: "Open X-Embodiment: Robotic Learning Datasets and RT-X Models"

[ICRA 2023] Intention Aware Robot Crowd Navigation with Attention-Based Interaction Graph

Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling

A recurrent attention module consisting of an LSTM cell which can query its own past cell states by the means of windowed multi-head attention. The formulas are derived from the BN-LSTM and the Transformer Network. The LARNN cell with attention can be easily used inside a loop on the cell state, just like any other RNN. (LARNN)

Transformers without Tears: Improving the Normalization of Self-Attention

Final Project for AI Wireless

BabyGPT: Build Your Own GPT Large Language Model from Scratch Pre-Training Generative Transformer Models: Building GPT from Scratch with a Step-by-Step Guide to Generative AI in PyTorch and Python

Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"