Llm create

@Logan M any idea how to do this? I tried several method but none worked.

16 comments

Tbh, I'm not sure what that create function does 😅

Does it create a dedicated model with that name on openai?

If so, you could just specify the model_name in the OpenAI/ChatOpenAI class right?

GGreg Tanaka

@Logan M I tried just putting in the "text-davinci-edit-001" string

but I get an error:

Cell In[67], line 5
2 print(nodes[i].text)
4 print("**GENERATED**")----> 5 response = global_index.query(nodes[i].text, similarity_top_k=5, mode="embedding", service_context = service_context, text_qa_template=QA_PROMPT) 6 display(Markdown(f"{response}")) 8 print("**SOURCES**")

File ~/anaconda3/envs/424b/lib/python3.11/site-packages/llama_index/indices/base.py:244, in BaseGPTIndex.query(self, query_str, mode, query_transform, use_async, **query_kwargs)
230 query_config = QueryConfig(
231 index_struct_type=self._index_struct.get_type(),
232 query_mode=mode_enum,
233 query_kwargs=query_kwargs,
234 )
235 query_runner = QueryRunner(
236 index_struct=self._index_struct,
237 service_context=self._service_context,
(...)
242 use_async=use_async,
243 )
--> 244 return query_runner.query(query_str)

File ~/anaconda3/envs/424b/lib/python3.11/site-packages/llama_index/indices/query/query_runner.py:341, in QueryRunner.query(self, query_str_or_bundle, index_id, level)
323 """Run query.
...
683 rbody, rcode, resp.data, rheaders, stream_error=stream_error
684 )
685 return resp

InvalidRequestError: Invalid URL (POST /v1/completions)

LLogan M

How did you create the llm predictor for the service context?

GGreg Tanaka

@Logan M llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-edit-001", max_tokens=1024))

LLogan M

Without llama index or anything, how would you normally query it? Does the URL change?

LLogan M

Just had a look at the openAI docs. I'm not sure if llamaindex or langchain is setup to handle edit requests 🤔

LLogan M

Yea, langchain is only setup for calling the completion endpoint https://github.com/hwchase17/langchain/blob/master/langchain/llms/openai.py#L248

GGreg Tanaka

@Logan M how about with Custom LLMs?

GGreg Tanaka

I tried doing something like this:
class CustomLLM(LLM):
model_name = "facebook/opt-iml-max-30b"
pipeline = pipeline("text-generation", model=model_name, device="cuda:0", model_kwargs={"torch_dtype":torch.bfloat16})

def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
prompt_length = len(prompt)
response = self.pipeline(prompt, max_new_tokens=num_output)[0]["generated_text"]

# only return newly generated tokens
return response[prompt_length:]

@property
def _identifying_params(self) -> Mapping[str, Any]:
return {"name_of_model": self.model_name}

@property
def _llm_type(self) -> str:
return "custom"

LLogan M

A custom pipeline could work! Instead of calling/running a local LLM, you could call the edit endpoint from openai 🤔

GGreg Tanaka

I just don't know how to modify the above code to do that

GGreg Tanaka

@Logan M where would I call this:

openai.Edit.create(
model="text-davinci-edit-001",
input="",
instruction="",
temperature=0.7,
top_p=1
)

LLogan M

Plain Text

class CustomLLM():
    def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
        result = openai.Edit.create(...)  # use the prompt as input?
        return result  # or parse the result here, not sure what this will actually be
    ... # the rest of the class is the same

LLogan M

Something like that. Just need to take the prompt and call openai with it, and return the resulting edited string

LLogan M

I'm not sure how well it will work... the prompt will also have llama indexs prompt template in it, along with your query and context

LLogan M

I'm not sure what the use case is, but I think this might be easier without llama index 😅

Add a reply

Find answers from the community

Llm create