Find answers from the community

Updated 3 months ago

I am using CTransformers and trying to

I am using CTransformers and trying to run it in the llamaindex pipeline [[response = qp.run(query_str="What is the correlation between survival and age?"]] but I am getting the following error:

AttributeError: 'CTransformers' object has no attribute 'set_callback_manager'

Can someone help me understand how to get a query pipeline working with my CTransformers local model?

import os
from langchain_community.llms import CTransformers
from llama_index.core import Settings


Set TRANSFORMERS_OFFLINE environment variable to 1

os.environ["TRANSFORMERS_OFFLINE"] = "1"

LLM_MODEL_NAME = ".cache/models/llama-2-7b-chat.Q5_K_M.gguf"
callback=[StreamingStdOutCallbackHandler()]

config = {'temperature': 0.0, 'context_length': 4096, 'stream': True}
llm = CTransformers(
model = LLM_MODEL_NAME,
model_type="gguf",
callbacks=callback,
config = config
)

Settings.llm = LangChainLLM(llm=llm)
L
g
14 comments
This is a langchain LLM right? need to wrap it
Plain Text
pip install llama-index-llms-langchain


Plain Text
from llama_index.llms.langchain import LangChainLLM

llm = LangChainLLM(CTransformers(...))
Yeah, I have it wrapped when I pass it here. The problem is when I call the query pipeline. llm = CTransformers(
model = LLM_MODEL_NAME,
model_type="gguf",
callbacks=callback,
config = config
)

Settings.llm = LangChainLLM(llm=llm)
You are sure you have Settings.llm = LangChainLLM(llm=llm) ? At the top of your code before setting up the query pipeline?
The error really indicates that thats not the case
The run query step is when I get the error above.
and you run this cell before you run any other cells?
Are you passing in the llm explicitly anywhere else?
I see the llm is a module in the query pipeline -- this also needs to be wrapped
ah, that worked. So even though I wrapped it in the settings, I also need to wrap it everywhere else? Understood. Thank you for your help! You are awesome and you for your prompt solutions!
Glad it works! And no problem :dotsHARDSTYLE:
Add a reply
Sign up and join the conversation on Discord