Find answers from the community

Updated 6 months ago

okay here's a more refined question, on

At a glance

okay here's a more refined question, on this page: https://docs.llamaindex.ai/en/stable/module_guides/querying/response_synthesizers/root.html#using-custom-prompt-templates-with-additional-variables

Where it calls TreeSummarize, what can I call instead that will allow for a compact response mode and a streaming response? I've delved into the code quite a bit but I'm not seeing anything out of the box

12 comments

LLogan M

Plain Text

from llama_index.core import get_response_synthesizer

synthesizer = get_response_synthesizer(response_mode="compact", llm=llm, ...)

LLogan M

I think thats what you wanted?

rredcoatwright

Hi, actually I've narrowed it down a bit further, take the example you just sent. When I do synthesizer.get_response(blah blah), how do I stream that response back? It seems to return a Generator object which I can iterate over but that isn't truly streaming, it's just waiting for the entire response and then printing it.

rredcoatwright

That's the last piece of my puzzle then I can deploy this rag pipeline to dev

LLogan M

I think this should work

Plain Text

synthesizer = get_response_synthesizer(response_mode="compact", llm=llm, streaming=True, ...)

response = synthesizer.get_response(blah blah)
for token in response:
    print(token, end="", flush=True)

rredcoatwright

Ah okay so you do just iterate over the gen object

rredcoatwright

okay, thank you!!

rredcoatwright

One quick question btw, does Llama have any sort of consulting or advising service my company can use? We're testing this out (obviously getting acquainted with it) but when we start putting things into production, I think we'd want to consult with experts.

LLogan M

We don't really have anything like that at the moment. The team is still quite small, and there isn't much bandwidth for contracting right now on top of everything we are working on 🙂

LLogan M

thankfully, the discord and github are pretty responsive

LLogan M

(as you can see, I'm out here in the trenches :dotsHARDSTYLE: )

rredcoatwright

no worries, I get it! early stage start up life

Add a reply