count-tokens-hf-datasets
PublicThis project shows how to derive the total number of training tokens from a large text dataset from ? datasets with Apache Beam and Dataflow.
Creat:2022-06-10T11:25:54
Update:2025-06-11T20:40:26
27
Stars
0
Stars Increase