Popular GitHub Repositories for Tokenizer

Unsupervised text tokenizer for Neural Network-based text generation.

Chinese version of GPT2 training code, using BERT tokenizer.

中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。

Underthesea - Vietnamese NLP Toolkit

Persian NLP Toolkit

Solves basic Russian NLP tasks, API for lower level Natasha projects

Unsupervised text tokenizer focused on computational efficiency

Jcseg is a light weight NLP framework developed with Java. Provide CJK and English segmentation based on MMSEG algorithm, With also keywords extraction, key sentence extraction, summary extraction implemented based on TEXTRANK algorithm. Jcseg had a build-in http server and search modules for lucene,solr,elasticsearch,opensearch

iBOT :robot:: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)

JTokkit is a Java tokenizer library designed for use with OpenAI models.

A Vietnamese natural language processing toolkit (NAACL 2018)

Implementation of MagViT2 Tokenizer in Pytorch

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

? Pretrained BERT model & WordPiece tokenizer trained on Korean Comments ??? ??? ??????? BERT ??? ????

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

A Japanese tokenizer based on recurrent neural networks

A Unified Tokenizer for Visual Generation and Understanding