I am using CTransformers and trying to

ggduck7959

I am using CTransformers and trying to run it in the llamaindex pipeline [[response = qp.run(query_str="What is the correlation between survival and age?"]] but I am getting the following error:

AttributeError: 'CTransformers' object has no attribute 'set_callback_manager'

Can someone help me understand how to get a query pipeline working with my CTransformers local model?

import os
from langchain_community.llms import CTransformers
from llama_index.core import Settings

Set TRANSFORMERS_OFFLINE environment variable to 1

os.environ["TRANSFORMERS_OFFLINE"] = "1"

LLM_MODEL_NAME = ".cache/models/llama-2-7b-chat.Q5_K_M.gguf"
callback=[StreamingStdOutCallbackHandler()]

config = {'temperature': 0.0, 'context_length': 4096, 'stream': True}
llm = CTransformers(
model = LLM_MODEL_NAME,
model_type="gguf",
callbacks=callback,
config = config
)

Settings.llm = LangChainLLM(llm=llm)

14 comments

LLogan M

This is a langchain LLM right? need to wrap it

LLogan M

Plain Text

pip install llama-index-llms-langchain

Plain Text

from llama_index.llms.langchain import LangChainLLM

llm = LangChainLLM(CTransformers(...))

ggduck7959

Yeah, I have it wrapped when I pass it here. The problem is when I call the query pipeline. llm = CTransformers(
model = LLM_MODEL_NAME,
model_type="gguf",
callbacks=callback,
config = config
)

Settings.llm = LangChainLLM(llm=llm)

ggduck7959

Attachment

LLogan M

You are sure you have Settings.llm = LangChainLLM(llm=llm) ? At the top of your code before setting up the query pipeline?

LLogan M

The error really indicates that thats not the case

ggduck7959

I do

Attachment

ggduck7959

I am following this guide, but just added a local model. https://docs.llamaindex.ai/en/stable/examples/pipeline/query_pipeline_pandas.html

ggduck7959

The run query step is when I get the error above.

LLogan M

and you run this cell before you run any other cells?

LLogan M

Are you passing in the llm explicitly anywhere else?

LLogan M

I see the llm is a module in the query pipeline -- this also needs to be wrapped

ggduck7959

ah, that worked. So even though I wrapped it in the settings, I also need to wrap it everywhere else? Understood. Thank you for your help! You are awesome and you for your prompt solutions!

LLogan M

Glad it works! And no problem :dotsHARDSTYLE:

Add a reply

Find answers from the community

I am using CTransformers and trying to

Set TRANSFORMERS_OFFLINE environment variable to 1