AIbase
Product LibraryTool Navigation

bpetokenizer

Public

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

Creat2024-05-27T03:05:35
Update2025-01-03T16:14:29
https://pypi.org/project/bpetokenizer/
3
Stars
0
Stars Increase

Related projects