Chonky is a multilingual paragraph segmentation model based on DistilBERT, fine-tuned on nearly 11 billion tokens from over 34 million Wikipedia articles. It can segment any natural language text into semantic chunks and supports 104 languages.

Natural Language ProcessingTransformersMultiple Languages

This is a Japanese pre - trained language model based on the Mistral 300M architecture. It is trained using the Wikipedia and cc100 datasets, and the byte - fallback technique in the SentencePiece tokenizer is adopted to suppress the generation of unknown words.

Natural Language ProcessingTransformers

A lightweight Dhivehi pre - trained model built on google/gemma-3-270m, trained on large - scale Dhivehi text data including news articles, Wikipedia content, and general web page texts, providing strong support for Dhivehi - related tasks.

Natural Language ProcessingTransformersOther

This is a quantized version based on the Pythia-70m model, specifically optimized for Wikipedia paragraph data, offering multiple quantization types to meet different resource requirements.

Natural Language ProcessingTransformersEnglish

A LoRA-adapted model based on google/gemma-2-9b, further pre-trained with Turkish Wikipedia data to enhance Turkish text processing capabilities

Natural Language ProcessingSafetensorsOther

A fastText classifier for evaluating the educational value of Japanese web pages, including two classifiers based on Wikipedia and LLM-generated data

Natural Language ProcessingFasttextJapanese

fermi-bert-1024 is a pre-trained BERT model specifically optimized for the nuclear energy field. This model is trained on data from Wikipedia, Books3, and the US Nuclear Regulatory Commission's ADAMS database, and it is specially optimized for handling technical terms and regulatory languages in the nuclear energy industry. It was trained on the Frontier supercomputer at Oak Ridge National Laboratory using 128 AMD MI250X GPUs for 10 hours, providing a solid foundation for nuclear energy applications.

Natural Language ProcessingTransformersEnglish

ElanMT-BT-ja-en is a Japanese-to-English translation model developed by the ELAN MITSUA Project/Abstract Engine, trained exclusively using open-license data and back-translated Wikipedia data.

Natural Language ProcessingTransformersMultiple Languages

ElanMT is a resource-efficient English-Japanese translation model developed by the ELAN MITSUA project/Abstract Engine. Despite relatively limited training resources, this model achieves performance comparable to current open-source translation models through back-translation and a newly constructed CC0 corpus. The entire training process uses only openly licensed data and Wikipedia back-translation data to ensure data purity.

Natural Language ProcessingTransformersMultiple Languages

The first groundbreaking large language model for Persian, with 13 billion parameters, trained on the Persian Wikipedia corpus, specifically designed for various natural language processing tasks.

Natural Language ProcessingTransformersOther

A text segmentation model fine-tuned based on the DistilBERT pre-trained model, specifically designed to determine whether two adjacent sentences come from the same Wikipedia article paragraph, enabling the automatic document chunking function.

Natural Language ProcessingTransformersEnglish

WikiChat is a large language model fine-tuned based on LLaMA-2 (7B). It prevents chatbots from generating hallucinations by anchoring a small number of samples on Wikipedia, significantly improving the accuracy and factuality of conversations.

Natural Language ProcessingTransformersEnglish

This is a RoBERTa base model based on Turkish. The pre - training data is sourced from Turkish Wikipedia, the Turkish OSCAR corpus, and some news websites.

Natural Language ProcessingTransformersturkish

A sentence transformation model based on BioBERT, specifically optimized for medical Wikipedia content, used for sentence similarity calculation

Natural Language ProcessingTransformers

The Bottleneck T5 Model is a text autoencoder based on the T5 architecture. It can encode text of up to 512 tokens into embedding vectors and reconstruct the original text from the embedding vectors. This model is trained on English Wikipedia data and supports semantic editing and text interpolation operations.

Natural Language ProcessingTransformersEnglish

GeoLM is a language model designed for detecting toponyms from sentences. It is pre-trained on global OpenStreetMap, WikiData, and Wikipedia data and fine-tuned on the GeoWebNews dataset.

Natural Language ProcessingTransformersEnglish

This is an unsupervised SimCSE model based on the BERT architecture, specifically optimized for Japanese text. This model is based on cl-tohoku/bert-base-japanese-v3 and is fine-tuned using the Japanese Wikipedia sentence dataset. It is mainly used for text similarity calculation and semantic feature extraction.

Natural Language ProcessingTransformersJapanese

A Japanese BigBird model pre-trained on Japanese Wikipedia, CC-100, and OSCAR datasets, suitable for long sequence processing tasks.

Natural Language ProcessingTransformersJapanese

A Wikipedia topic model built on the BERTopic framework, capable of extracting interpretable topic structures from massive text data

Natural Language ProcessingBertopic

This is a Japanese named entity recognition model built based on Chapter 6 of "Introduction to Large Language Models". It uses the BERT-base-japanese-v3 architecture and is fine-tuned on the Wikipedia dataset. It is specifically designed for named entity recognition tasks in Japanese text.

Natural Language ProcessingTransformersJapanese