Tokenizer

Just load the tokenizer for a non-gguf version of the model

1) when you say "tokenizer", which files you are referring from below screenshot. 2) I am building this code for on-edge application. Is there any way we can bundle tokenizer file(s) with the software so user don't have to download the tokenizer since customer's machine will not be connected to the internet. Thanks.

You can just download it locally and point it towards the file path

AutoTokenizer will look for the tokenizer* files from there (and maybe also config.json)

Trying to provide local path but it is throwing error. Can you provide correct command? Thanks.

I put quotes arounds the path directory strings and this error went away. But how to make sure this command is working? Thanks.