Llms

ttanaysai

If index was created with one llm model, and if you load that index with another llm model, would it work?
I have some indexes that are mixed up.

Directly looking at the json files, there seems to be no way of knowing which llm model they correspond to.
I tried to load those indexes, and tried things like index._service_context.llm_predictor.get_llm_metadata() etc, but they dont seem to indicate which LLM the index was created with.

On a prior thread there, it was suggested that I could define the llm and create a custom service object with that llm and load it into the service object while loading index from storage
Like index._service_context.llm_predictor.get_llm_metadata()

While I am doing that, i see error like below (likely because this index was created with another llm than the current one)

File "/home/paperspace/wynk/wynkenv/lib/python3.9/site-packages/torch/nn/functional.py", line 2515, in layer_norm
return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

4 comments

LLogan M

Indexes are LLM agnostic, it doesn't matter which LLM is used when loading/using the index

What DOES matter is the separate embedding model, since the query embeddings should match the embeddings in the index

LLogan M

The error below is coming from how you loaded your model

LLogan M

I think you have to do model = model.half() ? I've seen this error before with huggingface/pytorch, google should be helpful here lol

ttanaysai

Thanks Logan.

Yeah, based on the info you shared on other thread for printing LLM name from the index, I realised the same.

Took me a while to figure this out but incase this helps anyone

1) index generated from one model can be used with another as long as the embedding model used in query and index are the same.
2) the LLM present in the service context when you load an index from storage, is the default LLM of llama index.
3) The LLM isn't stored as meta data in index.
4) If you want to use the same LLM you while querying, as you used for index, create that LLM, create a service context with it, and pass that service context as Param while loading the index from storage.

Thanks @Logan M for answering my many dumb question and helping me figure these out across multiple threads.

Cheers

Add a reply

Find answers from the community

Llms