llama_parse/examples/demo_advanced.ipynb...

sscottyakc

Trying to replicate the LLamaParse example here: https://github.com/run-llama/llama_parse/blob/main/examples/demo_advanced.ipynb

But instead of OpenAI trying to use LM Studio on my local machine:
from llama_index.embeddings.huggingface import HuggingFaceEmbedding


embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
llm = OpenAI(
    api_key="NULL",
    api_base="http://localhost:1234/v1",
    temperature=0.2,
)

Settings.llm = llm
Settings.embed_model = embed_model

node_parser = MarkdownElementNodeParser(llm=llm, num_workers=8)

...

sub_query_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=query_engine_tools,
    llm=llm,
    use_async=True,
)

response = sub_query_engine.query(
    "Which fund has the smallest minimum investment size?"
)

The api call is received by the server, but when the response is sent back, Llama Index throws the below error:

ValueError: Expected tool_calls in ai_message.additional_kwargs, but none found.

Anyone experience this?

8 comments

LLogan M

Don't use OpenAI if you aren't using OpenAI, use OpenAILike

Plain Text

pip install llama-index-llms-openai-like

Plain Text

from llama_index.llms.open_like import OpenAILike

llm = OpenAILike(api_key="NULL", api_base="...", is_chat_model=True)

LLogan M

Otherwise, it thinks its a function calling model, and is trying to use the function calling API (which won't exist)

sscottyakc

Thanks! I tried to use the OpenAILike llm with the markdown node parse


llm = OpenAILike(
    api_key="NULL",
    api_base="http://localhost:1234/v1",
    temperature=0.1,
    is_chat_model=False
)

node_parser = MarkdownElementNodeParser(llm=llm, num_workers=8)

But I get the below, when trying to define node_parser:

ValidationError: 1 validation error for MarkdownElementNodeParser
llm
Can't instantiate abstract class LLM with abstract methods achat, acomplete, astream_chat, astream_complete, chat, complete, metadata, stream_chat, stream_complete (type=type_error)

LLogan M

how did you import it?

LLogan M

Seems like there is a mix of legacy and non-legacy imports here

sscottyakc

I had to import OpenAILike from legacy module, I get below when trying to import from the main package.

ModuleNotFoundError: No module named 'llama_index.llms.open_like'

Is there another llama-index version I should be using? I'm on 0.10.27

LLogan M

pip install llama-index-llms-openai-like

sscottyakc

Awesome, seems to be working now. You're the man, thanks!

Add a reply

Find answers from the community

llama_parse/examples/demo_advanced.ipynb...