Hi

If you are running without internet access, you'll need to have an LLM model and embedding model already downloaded to your machine somehow

so thats step #1 😅

I have it. This error appears, even before I create classes from llama_index.
It appears right on import llama_index
Because something wants to access the openai api from the __init__.py file.

It appeared after I upgraded from version 0.6.12 to 0.7.20

it's because tiktoken (the tokenizer used on the hood to count tokens, etc.) is trying to be downloaded. You might have to look up how to pre-download and cache this tokenizer

But I don't want to use it. I have another tokenizer

Maybe you know how to do it? And where should I save it so llama-index can take it?

Yea it's pretty baked into the codebase for counting tokens during chunking and other parts of llama-index. Kind of annoying, but it's tech-debt right now

Yea I have no idea lol I'd have to read their source code probably. A quick google search didn't turn up much

looks like the have an env var https://github.com/openai/tiktoken/blob/5d970c1100d3210b42497203d6b5c1e30cfda6cb/tiktoken/load.py#L30

for the location

There are two files you need here https://github.com/openai/tiktoken/blob/5d970c1100d3210b42497203d6b5c1e30cfda6cb/tiktoken_ext/openai_public.py#L12

And it looks for a folder in the cache dir by using a sha1 hash of the URL? lol weird

О! Thank you!!! I'll look into that 😢

I think those links should get you where you need to go! 🙏