Popular GitHub Repositories for Optimizers

??? 60+ Implementations/tutorials of deep learning papers with side-by-side notes ?; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), ? reinforcement learning (ppo, dqn), capsnet, distillation, ... ?

? Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

Nx-powered Neural Networks

Instantly improve your training performance of TensorFlow models with just 2 lines of code!

Toy implementations of some popular ML optimizers using Python/JAX

Code for "Accelerating Training with Neuron Interaction and Nowcasting Networks" [to appear at ICLR 2025]

This repository is a meticulously assembled anthology of optimization algorithms, meticulously implemented in PyTorch, designed to cater to the diverse needs of the machine learning research community.

MultiModelOptimizer: A Hierarchical Parameter Synchronization Approach for Joint Training of Multiple Transformer Models

dm-haiku implementation of hyperbolic neural networks

JAX implementation of 'Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training'