Popular GitHub Repositories for Multilingual Tokenization

? Industrial-strength Natural Language Processing (NLP) in Python

? Token-Oriented Object Notation – JSON for LLMs at half the token cost

Convert ebooks to audiobooks with chapters and metadata using dynamic AI models and voice cloning. Supports 1,107+ languages!

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

CodeGeeX: An Open Multilingual Code Generation Model (KDD 2023)

vits2 backbone with multilingual-bert

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

CodeGeeX2: A More Powerful Multilingual Code Generation Model

Multilingual Voice Understanding Model

High-quality multi-lingual text-to-speech library by MyShell.ai. Support English, Spanish, French, Chinese, Japanese and Korean.

GLM-4 series: Open Multilingual Multimodal Chat LMs | 开源多语言多模态对话模型

An Open Source Python alternative to NotebookLM's podcast feature: Transforming Multimodal Content into Captivating Multilingual Audio Conversations with GenAI

Gradio WebUI for creators and developers, featuring key TTS (Edge-TTS, kokoro) and zero-shot Voice Cloning (E2E, F5-TTS, CosyVoice), with Whisper audio processing, RVC voice changer, YouTube download, UVR5 vocal isolation, and multilingual translation.

??AutoDev: The AI-powered coding wizard（AI 驱动编程助手）with multilingual support ?, auto code generation ??, and a helpful bug-slaying assistant ?! Customizable prompts ? and a magic Auto Dev/Testing/Document/Agent feature ? included! ?

LightSeq: A High Performance Library for Sequence Processing and Generation

? 500+ handpicked Nano Banana Pro prompts with images, multilingual support, and instant gallery preview. Open-source prompt engineering library

Kandinsky 2 — multilingual text2image latent diffusion model

Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

MARY TTS -- an open-source, multilingual text-to-speech synthesis system written in pure java