gigant
/

bytes-tokenizer

Model card Files Files and versions

bytes-tokenizer / README.md

gigant's picture

Update README.md

9394d37 verified 4 months ago

|

history blame contribute delete

117 Bytes

metadata

license: mit

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file(tokenizer_path)