Popular GitHub Repositories for Tokens

YAYI 2 是中科闻歌研发的新一代开源大语言模型，采用了超过 2 万亿 Tokens 的高质量、多语言语料进行预训练。(Repo for YaYi 2 Chinese LLMs)

[ICLR 2025] SOTA discrete acoustic codec models with 40/75 tokens per second for audio language modeling

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Train a 1B LLM with 1T tokens from scratch by personal

Pytorch implementation of "All Tokens Matter: Token Labeling for Training Better Vision Transformers"

Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718

??Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy

Unofficial implementation of Google's FNet: Mixing Tokens with Fourier Transforms

? GUI for editing LLM vector embeddings. No more blind chunking. Upload content in any file extension, join and split chunks, edit metadata and embedding tokens + remove stop-words and punctuation with one click, add images, and download in .veml to share it with your team.

An implementation of a full named-entity evaluation metrics based on SemEval'13 Task 9 - not at tag/token level but considering all the tokens that are part of the named-entity

? Pretrained BigBird Model for Korean (up to 4096 tokens)

PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech

Implementation of TiTok, proposed by Bytedance in "An Image is Worth 32 Tokens for Reconstruction and Generation"

TTS for pitch-accented language. Korean dialect DB.

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

BERT classification model for processing texts longer than 512 tokens. Text is first divided into smaller chunks and after feeding them to BERT, intermediate results are pooled. The implementation allows fine-tuning.

NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.

Implementation of paper "Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on Compressed Spatial Tokens"

Turn a stream of token into a parsable JSON object as soon as possible. Enable Streaming UI for AI app based on LLM.