Popular GitHub Repositories for Tiktoken

ChatGPT Java SDK支持流式输出、Gpt插件、联网。支持OpenAI官方所有接口。ChatGPT的Java客户端。OpenAI GPT-3.5-Turb GPT-4 Api Client for Java

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

Ready-made tokenizer library for working with GPT and tiktoken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

OpenShield is a new generation security layer for AI models

This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

? GPT token estimation and context size utilities without a full tokenizer

A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.

MLX.zig: Lightweight Zig language bindings for Apple's MLX framework, enabling efficient machine learning directly on Apple Silicon with zero external dependencies.

Predict the exact openai token usage of functions

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

A demo of expanding the vocabulary of the Llama3 model, applicable to other vocabularies that use TikToken as well.

Extracts Google Sheets to JSONL for fine-tuning, estimates task costs with tiktoken.

console based game based on a llm

提取YouTube视频内容，将其作为知识输入到任何大型语言模型中

Fast & efficient BPE tokenizer written in C & python for LLM tranining

This project is part of Generative AI Nanodegree by Udacity