Find answers from the community

Updated 4 months ago

i tried to set this

At a glance

A community member is trying to set up a service context for a language model, but is encountering an error where the requested tokens exceed the context window. The community member tried setting the chunk_size_limit to 3000, but the error persists. Another community member suggests that the community member should try setting the context_window instead. The community member tries this, but still gets the same error.

ddonvito

i tried to set this

Plain Text

service_context = ServiceContext.from_defaults(llm='local', chunk_size_limit=3000)

but I am still getting this error using llama2-13B, the default one

Plain Text

File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/llama_cpp/llama.py", line 900, in _create_completion
    raise ValueError(
ValueError: Requested tokens (3993) exceed context window of 3900

Any ideas what I am doing wrongly?

3 comments

LLogan M

I can't remember if you figured this out already, but you actually want to set context_window

ddonvito

ok i'll try that. thanks!

ddonvito

i still get the same error. using this..

Plain Text

ServiceContext.from_defaults(llm='local', chunk_size_limit=1024, context_window=3000)

Add a reply