Popular GitHub Repositories for Sentence Piece Tokenizer

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

? Scalable embedding, reasoning, ranking for images and sentences with CLIP

? All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows

Unsupervised text tokenizer for Neural Network-based text generation.

Retrieval and Retrieval-augmented LLMs

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

Chinese version of GPT2 training code, using BERT tokenizer.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

Venom is a high-performance system developed with JavaScript to create a bot for WhatsApp, support for creating any interaction, such as customer service, media sending, sentence recognition based on artificial intelligence and all types of design architecture for WhatsApp.

text2vec, text to vector. 文本向量表征工具，把文本转化为向量矩阵，实现了Word2Vec、RankBM25、Sentence-BERT、CoSENT等文本表征、文本相似度计算模型，开箱即用。

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821

Top2Vec learns jointly embedded topic, document and word vectors.

? Accelerate inference and training of ? Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Efficient few-shot learning with Sentence Transformers

A curated list of pretrained sentence and word embedding models

Large Concept Models: Language modeling in a sentence representation space

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

中文长文本分类、短句子分类、多标签分类、两句子相似度（Chinese Text Classification of Keras NLP, multi-label classify, or sentence classify, long or short），字词句向量嵌入层（embeddings）和网络层（graph）构建基类，FastText，TextCNN，CharCNN，TextRNN, RCNN, DCNN, DPCNN, VDCNN, CRNN, Bert, Xlnet, Albert, Attention, DeepMoji, HAN, 胶囊网络-CapsuleNet, Transformer-encode, Seq2seq, SWEM, LEAM, TextGCN

jiant is an nlp toolkit