Find answers from the community

Updated 2 months ago

Custom llm

Hi! When I am using a custom LLM provider from the list of AI providers of Lanchain. How can I send custom parameters via the retrieve method? For example:

service_context = ServiceContext.from_defaults(llm=llm)
index = VectorStoreIndex.from_documents(all_docs, service_context=service_context)
retriver = index.as_retriever()
retriver.retrieve("what is the pricing")[0].node.text

I need to send in many parameters to my llm
L
m
3 comments
Retrieval does not use the LLM, it uses the embed model

Maybe more info would help find a solution for you πŸ‘
I am calling a LLM Provider, via Langchain (CerebriumAI) and for my model endpoint I need to send a prompt and the top vector that would have been retrieved from the vector store. I have the following code snippet:

llm = CerebriumAI(endpoint_url="https://run.cerebrium.ai/v2/p-ed25ab21/finetuned-lama/predict")
service_context = ServiceContext.from_defaults(llm=llm)
index = VectorStoreIndex.from_documents(all_docs, service_context=service_context)
query_engine = index.as_query_engine()
response = query_engine.query("Can you give me a code snippet of what deploying a hugging face model would look like?")

What I would like to send the Cerebrium endpoint is the following payload:
{
"prompt": "Can you give me a code snippet of what deploying a hugging face model would look like?"
"input": <Vector from vector store based on prompt"
}
Shouldn't the lamgchain LLM class itself format it for you like that?

There's no way to configure it to be passed like that, unless you implement a customLLM class
Add a reply
Sign up and join the conversation on Discord