Top k

SSeaCat

Hi! Is there any way to see how big text nodes that will be sent as context? I need to know in advance because in some cases, when similarity_top_k is 5, I don't want the exception to be thrown because text nodes are too big, then I would prefer to search with similarity_top_k 4 or 3.

25 comments

LLogan M

Llama index should be handling any size of retrieved context though?

You can probably implement a custom node postprocessor to check this though
https://gpt-index.readthedocs.io/en/stable/core_modules/query_modules/node_postprocessors/usage_pattern.html#custom-node-postprocessor

SSeaCat

@Logan M Well, I thought the same but I already had this exception, not sure why when I set top_k = 5 and it sent 5 chunks of ~19,000 symbols each not sure why

LLogan M

How are you using llama index? I can set the top k to 100 and it works fine 😅

SSeaCat

Plain Text

query_engine = index.as_chat_engine(verbose=True,chat_mode="context",
                similarity_top_k=5,
                system_prompt=prepared_system_prompt)

SSeaCat

actually, I have never hit this issue until yersterday

LLogan M

Ohhh you are using chat engine

SSeaCat

yes, is that bad?

LLogan M

Nah it just works a little differently

LLogan M

Yea using node postprocessors is probably the best solution

SSeaCat

Yeah, I see. I doesn't look complex but not sure how actually I could use it

SSeaCat

should I pass my postprocessor as a param?

LLogan M

Yea, like this

as_chat_engine(..., node_postprocessors=[...])

SSeaCat

cool, I see. Let me try it. Thanks!

SSeaCat

Hmm the example is not working. my version is 7.19, the code is the same but it can't find from llama_index.indices.postprocessor.base import BaseNodePostprocessor

SSeaCat

Ok, it's in .types not in .base

SSeaCat

@Logan M should I instantiate it or just pass a class name?

SSeaCat

@Logan M this is the code but the nodepostrpocessor is never hit, not sure why

Plain Text

query_engine = index.as_chat_engine(verbose=True,chat_mode="context",
                similarity_top_k=5,
                system_prompt=prepared_system_prompt, node_postprocessors=[CustomPostprocessor()])

LLogan M

What version of llamaindex do you have? I know this was added somewhat recently 🤔

SSeaCat

0.7.19

LLogan M

Ohhhh yea it won't be in that version 😅

SSeaCat

which min version should I move to to make it work?

LLogan M

It was added in 0.8.1

Currently we are on 0.8.19 though

https://gpt-index.readthedocs.io/en/stable/development/changelog.html#id47

SSeaCat

yeah I see. will I have to do a lot of fixes if moving to the latest version?