Find answers from the community

Updated 8 months ago

Hi! I want to summarize some documents.

At a glance
Hi! I want to summarize some documents. I remember in LangChain there were some stuffs like Refine, Map_Reduce etc. What type of strategies for summarization can LlamaIndex offer?

I found a section called Response Synthesizer in the official docs. Is it the right place to learn about summarization strategies in LlamaIndex?
L
p
26 comments
Probably youd want to use TreeSummarize
Plain Text
from llama_index.core.response_synthesizers import TreeSummarize

summarizer = TreeSummarize(llm=llm, use_async=True)

response = summarizer.get_response("Summarize the text", ["text1", "text2", ...])
@Logan M But what is it and why TreeSummarize and not another summarizer? :/
Its the only summarizer

It builds a bottom-up tree of summaries (i.e. summarize pairs of chunks, and then summarize those summaries, until you have one summary)
this is an explanation of all synthesizers
Great! So it summarizes the whole document autoregressively, by summarizing chunks + previous summary to get a final summary. Is my understanding correct?
essentially, but also as a tree 😅

Example

chunk1, chunk2, chunk3, chunk4

Summarize chunk1 and chunk2 = summary1
Summarize chunk3 and chunk4 = summary2

summarize summary1 and summary2 = summary 3

No more pairs, we've reached the root of the tree, return summary 3
Thank you very much @Logan M ! I swear I prefer LlamaIndex over Langchain just because of you (great community support) 😂
Wish more companies had the same support 👍
@Logan M When I use TreeSummarize class to summarize a list of my docs, I'm getting the following error:

Plain Text
ValidationError: 1 validation error for SummaryAndThemes
__root__
  Unterminated string starting at: line 1 column 618 (char 617) (type=value_error.jsondecode; msg=Unterminated string starting at; doc={"summary":"The comments reflect concerns about the rising cost of living, uncertainties about the future, challenges in coping with inflation, and the need for more support from the government. Many individuals express a lack of confidence in the current financial assistance provided and worry about job security and healthcare costs. There is a general sentiment of appreciation for the government's efforts but a desire for more substantial and long-term solutions to address the ongoing economic challenges.","themes":["Rising cost of living","Uncertainty about the future","Challenges in coping with inflation","Need; pos=617; lineno=1; colno=618)


Is it because of the prompt I'm using as a first argument in summarizer.get_response ?

I'm doing this:

Plain Text
class SummaryAndThemes(BaseModel):
    """Data model for a summary and themes."""

    summary: str
    themes: List[str]

summarizer = TreeSummarize(verbose=True, output_cls=SummaryAndThemes)

response = summarizer.get_response("""These are user comments, separated by new lines. \
Summarize them and list down key themes mentioned in these comments. 
""", 
    comment_chunks # each chunk contain ~4096 tokens (I'm using gpt-4o)
    )
it looks like the LLM ran out of space when generating the output class 👀
Might have to reduce the number of chunks you are passing in. Or try increasing max_tokens (although by default it is none, which for openai means to generate as much as there is room for)
Ah wait, so the sum of lengths of all chunks must be less than 128k? I thought If length of each individual chunk is less than 128k, then I’m safe 😂
hmm… It can’t be, actually.

There are 5 chunks in my list, each ~4096 tokens (so around 20k in total). Even with generated summaries, it should not go beyond 128k, isn’t it?
Hmmm, well really, it should be handling it. Maybe try setting max tokens llm = OpenAI(..., max_tokens=1024)
@Logan M Nah, didn't work 😦
mmm really? Looking at the code that should have definitely worked

You did something like this?
Plain Text
llm = OpenAI(model="gpt-3.5-turbo", max_tokens=1024)
summarizer = TreeSummarize(llm=llm, output_cls=SummaryAndThemes)
@Logan M I'm using gpt-4o model
yea same thing, but the above is what you had? (minus the model name)
@Logan M Yeah, my code is the same as yours
I tried to decrease chunk size up to 512, but still no luck. I think it is a problem with JSON parsing on OpenAI’s side, but not even sure how to debug
I think I know what’s the problem. I remember I had the similar error before while using PydanticProgram. Will try tomorrow and let you know If it works or not
tl;dr: I might have empty strings in my list e.g.: [“chunk1”, “chunk2”, “”, “chunk3”] , these empty strings used to cause the problem in PydanticProgram. Will check tomorrow
@Logan M Hi! I think we must explicitly send an llm argument to TreeSummarize . Even though I set Settings.llm=OpenAI(model="gpt-4o") beforehand, I am having this error:

Plain Text
BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 16385 tokens. However, you requested 16543 tokens (15446 in the messages, 73 in the functions, and 1024 in the completion). Please reduce the length of the messages, functions, or completion.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}


I think it uses gpt-3.5 . That's weird though, because yesterday I was getting a different error.

Anyways, I've initialized TreeSummarize object like this (explicitly passing the llm):

Plain Text
summarizer = TreeSummarize(verbose=True, output_cls=SummaryAndThemes, llm=Settings.llm)


and ran the summarizer.get_response function 10 times for testing, and not getting the error.
Hmm weird, it should be pulling from Settings if you dont pass it in
Glad it works now!
Add a reply
Sign up and join the conversation on Discord