Find answers from the community

Updated 9 months ago

getting this error when using vllm

getting this error when using vllm server
Attachment
image.png
d
L
22 comments
wait it works fine if not using openai server
Yea vllm server is for non-openai api

There is an OpenAILike llm class that works with any openai-like API

Should probably alias this
llama-index intergration wonders better than openai class and langchain
but can you pls combine pydantic funcitons with dataset generator lol
i cant figure it out
What are you trying to do? πŸ‘€
Everything!
Mostly have an llm itterate through a bunch of documents to label and classify stuff into the json. Oh and go through some transcripts to make a multurn dataset in sharegpt format lol
There is a llm.structured_predict() function that is useful for lower level structured outputs (with open source though this is always tricky)
@Logan M would that work with gpt4?
looks like its in the legacy. Do you know which version?
Attachment
image.png
Yea it'll work with gpt-4, or any llm really (openai models are typically the best though)
Lemme link the source code
Aww hail king Logan
Every LLM inherits from that base class where that function is defined πŸ‘
@Logan M wait sorry last question for the day. For the hyde documents is it gpt3.5 creating the hyde document. I wondering if different llms can improve hyde creation. If not ignore lol
oh shit but ughhh more experiments
@Logan M got it to work lol
Plain Text
for idx, node in enumerate(nodes):
    train_dataset_generator = DatasetGenerator(
        [node],
        llm=llm2,
        question_gen_query=question_gen_query,
        show_progress=True,

    )
    node_questions_0 = train_dataset_generator.generate_questions_from_nodes(num=2)
    for question in tqdm(node_questions_0):  
        prompt = PromptTemplate(question)
        print(f"[Node {idx}] Generated questions:\n {node_questions_0}")
        output = llm2.structured_predict(output_cls=Document, prompt=prompt)
        print(question)
o a JSON string
        json_output = json.dumps(jsonable_encoder(output), ensure_ascii=False)
        
        # Open the file in append mode to add the new entry
        with open('output.json', 'a', encoding='utf-8') as f:
            if not first_entry:
                f.write(',')  # Add a comma before the next entry if it's not the first one
            else:
                first_entry = False  # Update the flag after the first entry
            f.write(json_output)  # Write the JSON string

# After all entries are written, close the list with a ']'
with open('output.json', 'a', encoding='utf-8') as f:
    f.write(']')
Unless there's a more efficent way
there a reason mistral large doesnt work dataset generator ?
What do you mean by doesn't work?
Add a reply
Sign up and join the conversation on Discord