Hey all! I'm working on using

eedhenry

Hey all! I'm working on using CustomLLMs for and when working through the tutorial here:https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html#example-using-a-custom-llm-model-advanced

I am meet with the following error:

Plain Text

  File "/root/.cache/pypoetry/virtualenvs/pa-MPXmvNdN-py3.11/lib/python3.11/site-packages/llama_index/indices/prompt_helper.py", line 117, in from_llm_metadata
    context_window = llm_metadata.context_window
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'property' object has no attribute 'context_window'

I poked around the LLM and CustomLLM classes and things look OK at cursory glance and before I started digging in I wanted to see if anyone else faced this problem?

22 comments

LLogan M

How did you setup your custom LLM?

eedhenry

Exactly as in the example

LLogan M

hmmm one sec

LLogan M

I copied the code sample from the docs exactly and it worked 😅

LLogan M

What llama-index version are you on?

eedhenry

Plain Text

(pa-py3.11) /rags/pa/backend # poetry show |grep llama
llama-index        0.9.11.post1 Interface between LLMs and your data

LLogan M

So, for reference, I copied the code sample exactly into a python file (custom_llm.py), then ran it -- python ./custom_llm.py

You did the same?

eedhenry

I created a template project from create-llama, dropped to backend dir, dropped into poetry shell, installed all deps, added a class for my custom model and tested. I also tested the generic OurLLM in the example. Same problem

LLogan M

So, not quite the same, got it 😅 let me check how that would work

eedhenry

Was going to try subclassing LLMdirectly, next, but wanted to ask before I got too in there 🙂

eedhenry

dropped back to llama_index==0.9.11 instead of post1 and no dice

LLogan M

yea, going to run create-llama and plop in the LLM -- will let you know how it goes 👍

eedhenry

Thanks @Logan M ! Let me know if I can help in any way!

LLogan M

soooo it also worked for me lol

eedhenry

lol well crap

eedhenry

Alright - I was dev'ing in a container loop - dropping that out...standby

LLogan M

created a fastapi backend with create-llama
copied the imports + OurLLM() class into backend/app/utils/index.py at the top
run poetry shell + poetry install + pip install "transformers[torch]" (last one was so I could use local embeddings)
updated the service context to be

Plain Text

service_context = ServiceContext.from_defaults(
    llm=OurLLM(),
    embed_model="local"
)

LLogan M

Attachment

eedhenry

ha, oy, I think I wasn't instantiating the class correctly and left off the () - time for more coffee.

LLogan M

Got my cup next to me already ☕

eedhenry

Thanks Logan! Apologies for wasting time 🥴

LLogan M

haha no worries, a good sanity check!

Add a reply

Find answers from the community

Hey all! I'm working on using