AIbase

gpt-tokenizer

Public

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

Creat2023-03-22T12:32:09
Update2025-03-26T03:51:14
https://gpt-tokenizer.dev
596
Stars
0
Stars Increase