Moonlight is a Mixture of Experts (MoE) model with a total of 16B parameters and 3B active parameters, trained based on the Muon optimizer using 5.7T tokens. This model improves the current Pareto frontier, achieving better performance with fewer training FLOPs and excelling in multiple benchmark tests.

Natural Language ProcessingTransformers

fermi-bert-1024 is a pre-trained BERT model specifically optimized for the nuclear energy field. This model is trained on data from Wikipedia, Books3, and the US Nuclear Regulatory Commission's ADAMS database, and it is specially optimized for handling technical terms and regulatory languages in the nuclear energy industry. It was trained on the Frontier supercomputer at Oak Ridge National Laboratory using 128 AMD MI250X GPUs for 10 hours, providing a solid foundation for nuclear energy applications.

Natural Language ProcessingTransformersEnglish