Hi I have question answering task

At a glance

Hi! I have question answering task. Currently I'm using a vector store loaded with knowledge base data to answer user questions.
But we also have template answeres for the most common questions. And they are more exact and more complete then the answers we generate using the vector store. So what I would like to do is figure out if we have a template answer that fits the question and if we do use the template to answer it. But if we don't have a template answer, query the vector index and and generate an answer based on that.
Now I'm not sore how to tackle this. How should I store the templates? Do I build a separate vector store from templates and make a semantic search on both and compare the vector similarity or use maybe use one of the routers? Does anyone have some tips how to tackle this?

4 comments

LLogan M

hmmm, yea storing the Q/A templates in a vector db and doing retrieval makes sesne. If the similarity is high enough, then don't even ask the LLM, just send the answer from the template

Plain Text

query_engine = index.as_query_engine(similarity_top_k=3)
retriever = index.as_retriever(similarity_top_k=1)
source_node = retriever.retrieve(query_str)[0]

if source_node.score > 0.9:
  # return answer from source_node.node.text
else:
  return query_engine.query(query_str)

AAndro_

I see, thats how you use retrievers. Makes sense. Thanks!
But it would be two separate indexes, one for knowledge base data and another for templates right?

LLogan M

yea that makes sense I think!

AAndro_

To update this thread. I tried this out. But the similarity score between knowledge base results and templates is really not that significant that I could use it with any certainty.
What I'll try next is create a list of templates with descriptions when each should be used and ask GPT to classify the question and decide whether it can be answered with one of the templates or not.
Happy to hear other ideas as well.

Add a reply

Find answers from the community

Hi I have question answering task