Popular GitHub Repositories for Tokenization

? Industrial-strength Natural Language Processing (NLP) in Python

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Unsupervised text tokenizer focused on computational efficiency

From anything to mesh like human artists. Official impl. of "MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenization"

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

A Thai word tokenization library using Deep Neural Network

[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.

BNLP is a natural language processing toolkit for Bengali Language.

[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.

Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.