bpetokenizer
Public(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)
(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)