Popular GitHub Repositories for Subword Tokenization

? Industrial-strength Natural Language Processing (NLP) in Python

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

A curated list of pretrained sentence and word embedding models

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE)

:zap: TensorFlowASR: Almost State-of-the-art Automatic Speech Recognition in Tensorflow 2. Supported languages that can use characters or subwords

Unsupervised text tokenizer focused on computational efficiency

From anything to mesh like human artists. Official impl. of "MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenization"

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

A Thai word tokenization library using Deep Neural Network

[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.

BNLP is a natural language processing toolkit for Bengali Language.

[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition