If I have 7 PDF s They re all in regards

bbmax

If I have 7 PDF's, They're all in regards to winning a business RFP, but 3 of them won and 3 of them loss. I want to pass them all into GPT and ask it to create a scoring rubric.

25 comments

bbmax

@Logan M like... only way I can think of doing this is have a list index that summarizes each document

bbmax

and then passes all of them into another list index to create a rubric.

LLogan M

you want to score them all at once? One document at a time?

bbmax

yeah, I'd like it to create a rubric based on the 3 won and 3 lost

bbmax

is that crazy

LLogan M

ohhh I see I see... hmm

You could use an output_cls across all six documents in a list index? Or summarize them first, and then apply an output_cls?

Something like

Plain Text

class RubricField(BaseModel):
    """An attribute that can be measured on a scale of 1-5. Should be a specific attribute for measuring success (or failure) on project proposals."""
    name: str
    description: str

class Rubric(BaseModel):
    """A list of RubricFields for evaluating project proposals."""
    fields: List[RubricField]

LLogan M

I'm kind of pulling that out of my ass lol it might need some tweaking

LLogan M

but it makes sense? Maybe?

bbmax

haha but how do I tell it that these 3 documents "won" and these 3 documents "lost" and we want to create a rubric based on those

bbmax

it's almost like a memory thing

LLogan M

Yea that was my second thought

bbmax

also just to make sure metadata in documents are never passed thru llm?

bbmax

like if I could tell openai, "All of the documents with metadata won"

LLogan M

You might have to do it one document at a time, and put the win/lose status in the prompt, and then aggregate the results?

You could even take the final list of rubric fields and ask the LLM to condense them into more general fields at the end too, if there's too many

bbmax

like a list index for each PDF, "This pdf lost the bid, can you create a rubric based on what values" and then do that for each one and then pass all of those in as an index?

LLogan M

yea! Something like that!

bbmax

hmm

bbmax

no bueno haha

LLogan M

lol is there an obvious issue with that? I feel like it could work. Might use a lot of tokens though

bbmax

I just feel like ChatGPT works so much better lmao

bbmax

like If I could load all of these 7 docs into chat gpt it'd do a better job than what I can build