How to load a model, that I've already

At a glance

The community members are discussing how to load a pre-loaded model, specifically an embedding model. The original post mentions that when using ServiceContext.from_defaults(), the local flag must be declared for the embed_model parameter. However, when trying to load the embedding model directly using HuggingFaceEmbedding(), the local flag does not seem to work, and removing it causes the model to be downloaded again.

The comments suggest that embed_model = HuggingFaceEmbedding(model_name="WhereIsAI/UAE-Large-V1") should work without the local flag, but there is a difference in the cache directories used by the two code paths. The community members discuss whether it's a problem to download the model twice, and whether it would be better to consistently use the local flag for both ServiceContext and HuggingFaceEmbedding. There is no explicitly marked answer, but the discussion suggests that the community members are trying to find the best way to handle loading and reusing pre-loaded models.

ppikachu8887867

How to load a model, that I've already loaded?

Plain Text

service_context = ServiceContext.from_defaults(llm=llm, embed_model="local:WhereIsAI/UAE-Large-V1")

In the above example, I must declare local flag, right?

But If I need just an embedding model, what should I write?

Plain Text

embed_model = HuggingFaceEmbedding(model_name="local:WhereIsAI/UAE-Large-V1")

the above does not work and If I remove local , it will download the model again

8 comments

LLogan M

embed_model = HuggingFaceEmbedding(model_name="WhereIsAI/UAE-Large-V1") should work. Whats the issue?

I see those two code paths use a slightly different cache dir though :PSadge:

ppikachu8887867

yeah, exactly

LLogan M

not a huge deal to download it twice though? 🙏

LLogan M

I can change that to be consistant though

ppikachu8887867

Sure, I can download no problem.

Just was wondering If I can re-use it somehow.

ppikachu8887867

I think it will be better to force to add the local flag whenever we are using the local models:

e.g.: embed_model = HuggingFaceEmbedding(model_name="local:WhereIsAI/UAE-Large-V1")

So it will follow the same pattern as ServiceContext, otherwise we can add another argument in ServiceContext and embedding classes, to differ the local models from apis

LLogan M

The local flag is really only there for the service context, to make it clear its downloading and using that model

For huggingface embeddings, this is implied. I don't think it should have the local string thing (although I guess no reason why it shouldn't handle it for consistency)

Ideally, both of these use the same cache dir. ServiceContext uses the same cache dir with a /models suffix for some reason

ppikachu8887867

Alright 👍

Add a reply

Find answers from the community

How to load a model, that I've already