LlamaIndex replace LLM Provider

At a glance

Hello, I am trying to run Fast API Python backend for fullstack application ( from create-llama command). But with Replaced default LLM Provider. I am trying to replace OpenAI llm with PaLM llm and OpenAI embeddings with palm embeddings and I got some issue. Listing is here: https://gist.github.com/OTR/2eeca8d7fa8d5087397a3f9944b6a0fb

24 comments

LLogan M

seems like a bug with palm embeddings, although I could have sworn we fixed that

LLogan M

what version of llama-index do you have?

AAndrew

I just checked out it is quite old version, 0.8.69.post2 . It was boundled with npm package from here: https://www.npmjs.com/package/create-llama

AAndrew

Also, I wrote there lines after ServiceContextIstantiating and previous error massage has gone, but another one is pulled out

AAndrew

After that little patch, it does successfully create an index from documents, but seems like fails to load it again: see error listing

AAndrew

https://gist.github.com/OTR/e38bf8fb25ae7b4625e8e54727349a64

LLogan M

Seems like the vector store didn't save correctly, for whatever reason. Never seen that before lol

LLogan M

maybe check the default__vector_store.json file, it should be a JSON with this structure

Attachment

AAndrew

looks fine

AAndrew

Attachment

LLogan M

ah not fine actually

LLogan M

the embedding dict should be Dict[str, List[float]]

LLogan M

Seems like palm embeddings might still be bugged, they are returning a single float...

AAndrew

Are you able to reproduce the bug or I need to provide additional information?

LLogan M

I don't have access to palm to test, but I would appreciate a PR if you have time!

Basically need to test that each of these methods are returning the proper types...
https://github.com/run-llama/llama_index/blob/41710721d23a35093963573128b18ccf20c5d757/llama_index/embeddings/google_palm.py#L51

AAndrew

I can share my API KEY

LLogan M

yea that works too, if you don't mind

LLogan M

got it 🙂

LLogan M

(deleted your message as well lol)

LLogan M

weird, I am unable to get it to work

Plain Text

>>> from llama_index.embeddings import GooglePaLMEmbedding
>>> embed_model = GooglePaLMEmbedding(api_key="....", model_name="models/embedding-gecko-001")
>>> out = embed_model.get_text_embedding("test")

Just returns a 503 error. Some locally, locally on a VPN, and on google colab 🤔

AAndrew

Try this KEY with text-bison-001 (PaLM 2). It should fit all of them,

AAndrew

Plain Text

llm=PaLM(
        api_key=os.getenv("GOOGLE_AI_API_KEY"),
        model_name="models/text-bison-001",
    )

LLogan M

similar error there 😅 I know google-palm isn't available in Canada, but usually using a VPN or google colab gets around that... but not anymore it seems? :PSadge:

LLogan M

I really recommend you test and make a PR yourself, it's really easy 🙂

Add a reply

Find answers from the community

LlamaIndex replace LLM Provider