Popular GitHub Repositories for Pretrain

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

[ICLR'23 Spotlight?] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.

从0到1构建一个MiniLLM (pretrain+sft+dpo实践中)

Firefly中文LLaMA-2大模型，支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型

An official implementation for " UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation"

Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)

GraphGen：利用知识驱动的合成数据生成增强大型语言模型的监督微调

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

Paper Lists, Notes and Slides, Focus on NLP. For summarization, please refer to https://github.com/xcfcode/Summarization-Papers

BERT-CCPoem is an BERT-based pre-trained model particularly for Chinese classical poetry

An Efficient Text-to-Image Generation Pretrain Pipeline

Bert-based models(BERT, MTB, CP) for relation extraction.

Tutorial to pretrain & fine-tune a ? Flax T5 model on a TPUv3-8 with GCP

MatDGL is a neural network package that allows researchers to train custom models for crystal modeling tasks. It aims to accelerate the research and application of material science.

turn bert pretrain checkpoint into saved model for a feature extracting demo in java

This repository uses pretrain BERT embeddings for transfer learning in QA domain