Popular GitHub Repositories for Korean Tokenizer

PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/Docker/Zotero

Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.

Unsupervised text tokenizer for Neural Network-based text generation.

Chinese version of GPT2 training code, using BERT tokenizer.

High-quality multi-lingual text-to-speech library by MyShell.ai. Support English, Spanish, French, Chinese, Japanese and Korean.

:stuck_out_tongue_closed_eyes: TensorFlowTTS: Real-Time State-of-the-art Speech Synthesis for Tensorflow 2 (supported including English, French, Korean, Chinese, German and Easy to adapt for other languages)

KoAlpaca: ??? ???? ???? ???? ???? (KoAlpaca: An open-source language model to understand Korean instructions)

中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。

Underthesea - Vietnamese NLP Toolkit

Python package for Korean natural language processing.

Korean BERT pre-trained cased (KoBERT)

Persian NLP Toolkit

Solves basic Russian NLP tasks, API for lower level Natasha projects

??? 개발자를 위한 북마크 모음집

KakaoBrain KoGPT (Korean Generative Pre-trained Transformer)

Unsupervised text tokenizer focused on computational efficiency

Jcseg is a light weight NLP framework developed with Java. Provide CJK and English segmentation based on MMSEG algorithm, With also keywords extraction, key sentence extraction, summary extraction implemented based on TEXTRANK algorithm. Jcseg had a build-in http server and search modules for lucene,solr,elasticsearch,opensearch

iBOT :robot:: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)

World's fastest ANPR / ALPR implementation for CPUs, GPUs, VPUs and NPUs using deep learning (Tensorflow, Tensorflow lite, TensorRT, OpenVX, OpenVINO). Multi-Charset (Latin, Korean, Chinese) & Multi-OS (Jetson, Android, Raspberry Pi, Linux, Windows) & Multi-Arch (ARM, x86).

JTokkit is a Java tokenizer library designed for use with OpenAI models.