Refresh

aalexnn

Hi, in our org the LLM api_key expires every hour. We load a bunch of indices in memory and then run queries on them. So for the first hour everything works. But once token expires we get 401. We refresh the api_key through Settings.llm and Settings.embed_model but it does not help - still getting 401 expired token when querying.

To fix this we tried to recreate query engines after the token is refreshed I can see that llm.api_key contains new token, but retriever._embed_model.api_key is old. Which I suspect causing this issue. From what I can see in code .as_query_engine factory method resolves llm but it does not resolve embed_model. Is there a reason for that?

What is the right way to refresh the api_key for existing loaded index / query engine?

4 comments

LLogan M

You can pass in the refreshed llm and embeddings model into the query engine

index.as_query_engine(llm=llm, embed_model=embed_model)

aalexnn

Thanks for getting back to me! I tried that, but when I pass embed_model I get TypeError:

llama_index.core.indices.document_summary.retrievers.DocumentSummaryIndexEmbeddingRetriever got multiple values for keyword argument "embed_model"

LLogan M

lamee

You can do either index._embed_model = embed_model or create the retriever and query engine on your own

Plain Text

from llama_index.core.indices.document_summary.retrievers import DocumentSummaryIndexEmbeddingRetriever

retriever = DocumentSummaryIndexEmbeddingRetriever(index, embed_model=self._embed_model)

from llama_index.core.query_engine import RetrieverQueryEngine

query_engine = RetrieverQueryEngine.from_args(retriever, llm=llm)

aalexnn

Thanks a lot! I prefer not to access private fields, the second suggestion to create query_engine manually works great!

Add a reply

Find answers from the community

Refresh