LlamaIndex

Log inLog into community

Find answers from the community

Updated 3 months ago

help getting output classes to work.

help getting output classes to work.

·

help getting output classes to work. Hoping that someone can take a peek at my code and give me some guidance as to how to get output classes to format the response. I'm getting key errors and attribute not found errors. In the prompt I instruct the LLM to generate a key value pair for its classification. How to debug this? thank you!

class TechnicalNoteClassifcations(Enum):
    CRITICAL = "Critical"
    NEW_FEATURE = "New feature"
    SOLUTION_PROVIDED = "Solution provided"
    INFORMATION_ONLY = "Information only"
class TechnicalNoteResponseData(BaseModel):
    classification: TechnicalNoteClassifcations
    summary: str
response = index.as_query_engine(
                    text_qa_template=qa_prompt_templates[item],
                    similarity_top_k=num_k,
                    output_cls=TechnicalNoteResponseData
                    ).query(f"{title}")

❱ 51 │   │   │   │   │   ).query(f"{title}")  
KeyError: 'classification'

t

b

24 comments

Was wondering if anyone has some experience getting output classes to work? I'd really like to figure out how to use pydantic to structure the response output.

hello!

I built this!!

humble brag / brag 🙂

w/ logan's help obviously

https://docs.llamaindex.ai/en/stable/examples/output_parsing/openai_pydantic_program.html

you shouldn't need to add it to your prompt, just passing in the output_cls should be enough, are you using an openai chat model?

Oh I see you're getting an error for classification, is there more to that error?

I wonder if it's because your TechnicalNoteclassifications is an Enum

Thanks @bmax I'm not sure. I'm not using it on chat model, I'm trying to get it working for a query engine. I wish I knew what the issue was. 😛

try not to use ENUM

how do I get the LLM to choose an option then? I want the LLM to make a selection from a set list of options...

what llm are you using?

openai

Here are the two things I'd try:

try just a list object of the strings
https://docs.pydantic.dev/latest/api/standard_library_types/#enum try to make it more engrained with pydantic
print TechnicalNoteResponseData.schema() and let me see what that looks like (with enum or not)

ok thanks @bmax I'll get that code working later today as I had moved on from it and I dont'remember what state it was in 😄 so many notebooks of learnings lol

I printed out the schema and re-ran the query_engine:

{'definitions': {'TechnicalNoteClassifications': {'description': 'An '
                                                                 'enumeration.',
                                                  'enum': ['Critical',
                                                           'New feature',
                                                           'Solution provided',
                                                           'Information only'],
                                                  'title': 'TechnicalNoteClassifications'}},
 'properties': {'classification': {'$ref': '#/definitions/TechnicalNoteClassifications'},
                'summary': {'title': 'Summary', 'type': 'string'}},
 'required': ['classification', 'summary'],
 'title': 'TechnicalNoteResponseData',
 'type': 'object'}

the response object doesn't have the attributes of the output class:


4 │   similarity_top_k=5,                                                                      │
│   5 │   output_cls=TechnicalNoteResponseData                                                     │
│   6 ).query("CVE-2023-36409 Microsoft Edge (Chromium-based) Information Disclosure Vulnerabi     │
│ ❱ 7 print(response.classification) 
AttributeError: 'Response' object has no attribute 'classification'

any thoughts?

I'm pretty sure that means OpenAI isn't returning that key

what happens if you print response/

ok. when I print response I get:

Classification: Information only

Summary: The post provides information about a vulnerability in Microsoft Edge (Chromium-based) related to CVE-2023-36409, which is an information disclosure vulnerability. The post states that the latest version of Microsoft Edge (Chromium-based) is no longer vulnerable to this issue. No specific solution or fix is mentioned in the post. The vulnerability is documented in the Security Update Guide.

so my question is, how can I extract the key value pairs from this output?

it looks like it worked!!

can you send whole notebook, i'm a bit confused on why it sent it in that format

the notebook is chaos 😄 the class definitions are near the top and cell where I run the query_engine is about half way down... looks like

index = vector_store_indicies['msrc_security_update']
response = index.as_query_engine(
    text_qa_template=qa_prompt_templates['msrc_security_update'],
    similarity_top_k=5,
    output_cls=TechnicalNoteResponseData
).query("CVE-2023-36409 Microsoft Edge (Chromium-based) Information Disclosure Vulnerability")
print(response)
#print(response.source_nodes[0].node.metadata)

removed all non related code

Add a reply

Sign up and join the conversation on Discord