Hey everyone,

GGero

Hey everyone,

I'm currently working with Query Pipelines and I'm encountering an issue when passing a list of reranked nodes to a synthesizer. The problem arises when the entire list of nodes gets cropped in the prompt. I'm wondering if anyone has any insights into what might be causing this issue.

Plain Text

user_prompt = PromptTemplate("{query}")
reranker = CohereRerank(model="rerank-multilingual-v3.0", api_key=api_key, top_n=3)
response_synthesis_prompt = PromptTemplate(
    response_synthesis_prompt_str,
)
llm = OpenAI(model="gpt-4-turbo", system_prompt=system_prompt)

Plain Text

#define qp
p = QueryPipeline(verbose=True)
p.add_modules(
    {   
        "user_prompt": user_prompt,
        "router_retriever": router_retriever,
        "kg_retriever": router_retriever,
        "reranker": reranker,
        "response_synthesis_prompt": response_synthesis_prompt,
        "response_synthesis_llm": llm,
    }
)

Plain Text

p.add_link("user_prompt", "router_retriever")
p.add_link("user_prompt", "reranker", dest_key="query_str")
p.add_link("router_retriever", "reranker", dest_key="nodes")

p.add_link("user_prompt", "kg_retriever")

p.add_link("user_prompt", "response_synthesis_prompt", dest_key="query_str")
p.add_link("reranker", "response_synthesis_prompt", dest_key="nodes")
p.add_link("kg_retriever", "response_synthesis_prompt", dest_key="kg_nodes")
p.add_link("response_synthesis_prompt", "response_synthesis_llm")

I suspect that the length of the reranked nodes list may be exceeding the prompt size limits, leading to it being truncated with ellipses (...). However, I'm unsure how to address this issue effectively.

If anyone has encountered a similar problem or has suggestions on how to resolve it, I would greatly appreciate any advice or guidance you can provide.

Thanks in advance for your help!

Attachment

3 comments

LLogan M

I'm not sure what you mean by truncated. Nothing here would ever truncate. Your screenshot also looks fine

The one thing that is kind sus here though is I would be using an actual response synthesizer though, instead of raw prompt and llm? But up to you.

Lastly, you can run your pipeline and inspect the outputs at each step as well
https://docs.llamaindex.ai/en/stable/module_guides/querying/pipeline/usage_pattern/#intermediate-outputs

GGero

You'll notice that the node.text is truncated with "..." However, the full NodesWithScore objects are longer than what's displayed.
Could you clarify what you mean by "raw prompt"? I'm a bit confused about that.

LLogan M

Ah I see now what you mean

Yea since you are just stuff node objects into a prompt, I have no idea how its string-ifying those nodes (seems like its just casting as string)

I would use an actual response synthesizer, or add some custom component to turn your nodes into a proper string before giving it to the prompt

Add a reply

Find answers from the community

Hey everyone,