Popular GitHub Repositories for Bpe

Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE)

Unsupervised text tokenizer focused on computational efficiency

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

Ready-made tokenizer library for working with GPT and tiktoken

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Swift BPE Encoder/Decoder for OpenAI GPT Models. A programmatic interface for tokenizing text for OpenAI ChatGPT API.

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Swift Tokenizer / BPE Encoder for Mistral 7B

Cross-lingual Language Model (XLM) pretraining and Model-Agnostic Meta-Learning (MAML) for fast adaptation of deep networks

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Sentiment-based classification for stock article title using PhoBert

Pretrained GPT2 Turkish model that is meant to be an entry point for fine-tuning on other texts.

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

Code for the paper "BPE stays on SCRIPT"

BPE Tokenizer implementations in C# for Anthropic, OpenAI LLM offerings

BPE Tokenizer for OpenAI's models

BPE tokenizer used for Dart/Flutter applications when calling ChatGPT APIs

n8n node for working with BPE Tokens with GPT in mind.

这个代码库用纯Python提供了一个清晰易懂的字节对编码（BPE）分词的实现。重点在于算法理解，而非运行速度。

讲解基于Transformer的架构