Guidance

kkj6i7qg2

Hello, I have a question about using the Guidance Pydantic Program, as documented here: https://docs.llamaindex.ai/en/latest/examples/output_parsing/guidance_pydantic_program.html . I have a query_engine defined and my question is how do I pass the program variable (as shown on the documentation page) to the query engine:

Plain Text

### program defined based on the documentation
program = GuidancePydanticProgram(
    output_cls=Result,
    prompt_template_str="Generate a response using the query asked as follows: {{query}}",
    guidance_llm=self.llm,
    verbose=True,
)

### standard query_engine defined, but how do I pass in the program variable or get the program to see the vector index?
query_engine = index.as_query_engine(
    similarity_top_k=2,
    node_postprocessors=[
        MetadataReplacementPostProcessor(target_metadata_key="window")
    ],
    output_cls=Result,
)
response = query_engine.query(query)

8 comments

WWhiteFang_Jr

Maybe this could help: https://docs.llamaindex.ai/en/latest/examples/output_parsing/guidance_sub_question.html

LLogan M

currently you can't pass the program to the query engine, you can only pass the output class

Plain Text

index.as_query_engine(output_cls=my_output_cls)

https://docs.llamaindex.ai/en/stable/examples/query_engine/pydantic_query_engine.html

kkj6i7qg2

Thanks @WhiteFang_Jr , the sub question approach looks promising. Do you know if sub-questions are supported using LlamaCpp or another local model llm like HuggingFaceLLM?

LLogan M

They are, but they might not be reliable. It's pretty hard to get open-source models to generate structured outputs reliably

the new zephyr-7B model seems a little promising. Otherwise you need like 30B+ parameters 😅

kkj6i7qg2

Thanks @Logan M. Trying out Zephyr. When the model's output is not reliable, is the expectation that LlamaIndex raises an exception and exits, like this (just want to confirm I don't have an error in my code):

Plain Text

raise OutputParserException(
llama_index.output_parsers.base.OutputParserException: Failed to parse pydantic object from guidance program

LLogan M

yea zephyr is probably the most reliable I've tested, but it can stll out errors as well

You could probably try the text completion program too though. It's a bit more simple under the hood imo

Plain Text

from llama_index.output_parsers import PydanticOutputParser

prompt_template_str = """\
Generate an example album, with an artist and a list of songs. \
Using the movie {movie_name} as inspiration.\
"""
program = LLMTextCompletionProgram.from_defaults(
    output_parser=PydanticOutputParser(Album),
    prompt_template_str=prompt_template_str,
    llm=llm,
    verbose=True,
)

kkj6i7qg2

Thanks @Logan M, I'm seeing this working well with Zephyr on some initial testing. I just put it in a try/except block and when there is a format error based on the LLM output, I set the Pydantic class values to None .

LLogan M

Neat fix!

Add a reply

Find answers from the community

Guidance