Is there a way to use

dddd_avila

Is there a way to use LMFormatEnforcerPydanticProgram in a query engine? I'm using a quantized LLama2-based model through LLamaCPP, and from there, I'm trying to use the SubQuestionQueryEngine like this

Plain Text

query_engine_tools = [
    QueryEngineTool(
        query_engine=index.as_query_engine(similarity_top_k=10, node_postprocessors=[meta_processor, reranker_procesor]),
        metadata=ToolMetadata(
            name=f"us_evidence_rules",
            description=f"Useful for queries about evidence rules of the US federal government",
        ),
    )
]

response_synthesizer = get_response_synthesizer(response_mode="compact")
query_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=query_engine_tools,
    response_synthesizer=response_synthesizer,
)

Nevertheless, I keep getting the same error due to the output parsing:

Plain Text

ValidationError: 1 validation error for SubQuestion
__root__
  SubQuestion expected dict not str (type=type_error)

5 comments

LLogan M

There isn't currently a way. The llm technicaly has a pydantic_program_mode kwarg that sets the type of program to use (text completion, guidance, llm format enforancer) -- but its only used if llm.structured_predict() is called, which the question generator doesn't use it

LLogan M

its been on my list to get to this, haven't got to it yet

dddd_avila

And can you think of a work around? Is there a way to use it inside a QueryPipeline?

LLogan M

you could design your own query pipeline using custom component yes

LLogan M

design your own way to generate sub questions, run those against their respective sub-query engines, do a final consolidation of responses

Add a reply

Find answers from the community

Is there a way to use