Hi! I want to summarize some documents.

At a glance

Hi! I want to summarize some documents. I remember in LangChain there were some stuffs like Refine, Map_Reduce etc. What type of strategies for summarization can LlamaIndex offer?

I found a section called Response Synthesizer in the official docs. Is it the right place to learn about summarization strategies in LlamaIndex?

26 comments

LLogan M

Probably youd want to use TreeSummarize

LLogan M

Plain Text

from llama_index.core.response_synthesizers import TreeSummarize

summarizer = TreeSummarize(llm=llm, use_async=True)

response = summarizer.get_response("Summarize the text", ["text1", "text2", ...])

ppikachu8887867

@Logan M But what is it and why TreeSummarize and not another summarizer? :/

LLogan M

Its the only summarizer

It builds a bottom-up tree of summaries (i.e. summarize pairs of chunks, and then summarize those summaries, until you have one summary)

LLogan M

https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/response_modes/?h=response+mode

LLogan M

this is an explanation of all synthesizers

ppikachu8887867

Great! So it summarizes the whole document autoregressively, by summarizing chunks + previous summary to get a final summary. Is my understanding correct?

LLogan M

essentially, but also as a tree 😅

Example

chunk1, chunk2, chunk3, chunk4

Summarize chunk1 and chunk2 = summary1
Summarize chunk3 and chunk4 = summary2

summarize summary1 and summary2 = summary 3

No more pairs, we've reached the root of the tree, return summary 3

ppikachu8887867

Thank you very much @Logan M ! I swear I prefer LlamaIndex over Langchain just because of you (great community support) 😂
Wish more companies had the same support 👍

ppikachu8887867

@Logan M When I use TreeSummarize class to summarize a list of my docs, I'm getting the following error:

Plain Text

ValidationError: 1 validation error for SummaryAndThemes
__root__
  Unterminated string starting at: line 1 column 618 (char 617) (type=value_error.jsondecode; msg=Unterminated string starting at; doc={"summary":"The comments reflect concerns about the rising cost of living, uncertainties about the future, challenges in coping with inflation, and the need for more support from the government. Many individuals express a lack of confidence in the current financial assistance provided and worry about job security and healthcare costs. There is a general sentiment of appreciation for the government's efforts but a desire for more substantial and long-term solutions to address the ongoing economic challenges.","themes":["Rising cost of living","Uncertainty about the future","Challenges in coping with inflation","Need; pos=617; lineno=1; colno=618)

Is it because of the prompt I'm using as a first argument in summarizer.get_response ?

I'm doing this:

Plain Text

class SummaryAndThemes(BaseModel):
    """Data model for a summary and themes."""

    summary: str
    themes: List[str]

summarizer = TreeSummarize(verbose=True, output_cls=SummaryAndThemes)

response = summarizer.get_response("""These are user comments, separated by new lines. \
Summarize them and list down key themes mentioned in these comments. 
""", 
    comment_chunks # each chunk contain ~4096 tokens (I'm using gpt-4o)
    )

LLogan M

it looks like the LLM ran out of space when generating the output class 👀

LLogan M

Might have to reduce the number of chunks you are passing in. Or try increasing max_tokens (although by default it is none, which for openai means to generate as much as there is room for)

ppikachu8887867

Ah wait, so the sum of lengths of all chunks must be less than 128k? I thought If length of each individual chunk is less than 128k, then I’m safe 😂

ppikachu8887867

hmm… It can’t be, actually.

There are 5 chunks in my list, each ~4096 tokens (so around 20k in total). Even with generated summaries, it should not go beyond 128k, isn’t it?

LLogan M

Hmmm, well really, it should be handling it. Maybe try setting max tokens llm = OpenAI(..., max_tokens=1024)

ppikachu8887867

@Logan M Nah, didn't work 😦

LLogan M

mmm really? Looking at the code that should have definitely worked

You did something like this?

Plain Text

llm = OpenAI(model="gpt-3.5-turbo", max_tokens=1024)
summarizer = TreeSummarize(llm=llm, output_cls=SummaryAndThemes)

ppikachu8887867

@Logan M I'm using gpt-4o model

LLogan M

yea same thing, but the above is what you had? (minus the model name)

ppikachu8887867

@Logan M Yeah, my code is the same as yours

ppikachu8887867

I tried to decrease chunk size up to 512, but still no luck. I think it is a problem with JSON parsing on OpenAI’s side, but not even sure how to debug

ppikachu8887867

I think I know what’s the problem. I remember I had the similar error before while using PydanticProgram. Will try tomorrow and let you know If it works or not

ppikachu8887867

tl;dr: I might have empty strings in my list e.g.: [“chunk1”, “chunk2”, “”, “chunk3”] , these empty strings used to cause the problem in PydanticProgram. Will check tomorrow

ppikachu8887867

@Logan M Hi! I think we must explicitly send an llm argument to TreeSummarize . Even though I set Settings.llm=OpenAI(model="gpt-4o") beforehand, I am having this error:

Plain Text

BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 16385 tokens. However, you requested 16543 tokens (15446 in the messages, 73 in the functions, and 1024 in the completion). Please reduce the length of the messages, functions, or completion.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

I think it uses gpt-3.5 . That's weird though, because yesterday I was getting a different error.

Anyways, I've initialized TreeSummarize object like this (explicitly passing the llm):

Plain Text

summarizer = TreeSummarize(verbose=True, output_cls=SummaryAndThemes, llm=Settings.llm)

and ran the summarizer.get_response function 10 times for testing, and not getting the error.

LLogan M

Hmm weird, it should be pulling from Settings if you dont pass it in

LLogan M

Glad it works now!

Add a reply

Find answers from the community

Hi! I want to summarize some documents.