Source Nodes

At a glance

The community members are discussing how to determine if the answer to a query was found in the context. They suggest using the fuzzy_citation tool from the LLama Hub or the structured_answer_filtering feature from LlamaIndex to extract the specific parts of the source nodes used to generate the response. However, there are some issues with the implementation, such as the response not containing the query_satisfied argument, and problems with the extracted parts of the source nodes being inaccurate or causing errors. The community members encourage further investigation and modification of the tools to address these issues.

Useful resources

SStéphane

Hi, is there a parameter in the response from a query_engine that would allow me to know if the LLM has decided that the answer was present in the context? I would like to display the source nodes used to answer the query but only if the answer to this query was found in the context.

5 comments

WWhiteFang_Jr

I think you can use: https://github.com/run-llama/llama-hub/tree/main/llama_hub/llama_packs/fuzzy_citation

This will help you to find the exact part in the source node which has been used to form the response

LLogan M

The above should work quite well.

If not, there is also this, which gets the LLM to write an answer and also decide if that answer satisfies the query
https://docs.llamaindex.ai/en/stable/examples/response_synthesizers/structured_refine.html

SStéphane

@Logan M

Plain Text

structured_answer_filtering

looks the most promising! Can it be used with a query engine directly though? My code looks like:

Plain Text

query_engine = index.as_query_engine(
            text_qa_template=qa_template, 
            response_mode="compact", 
            structured_answer_filtering=True)
response = query_engine.query(prompt)
print(response)

but the response variable doesn’t seem to contain the

Plain Text

query_satisfied

argument.

SStéphane

do you know if fuzzy_citation needs a specific tokenizer, text splitter or llm in order to work properly? I’m using

Plain Text

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en", max_length=512)

for embeddings and zephyr-7b-beta for llm, but the extracted parts of the source node used for the response are always a bit off (and for some prompts, I get an

Plain Text

IndexError: list index out of range

error)

LLogan M

It doesn't use any specific tokenizer. But I encourage you to take a look at what it's doing and modify as you see fit

Add a reply

Find answers from the community

Source Nodes