Extracting Information from Websites to Answer Queries

The post asks about the best way to extract information from a website and respond to queries about the website and its links. The comments provide some suggestions:

A community member recommends using a library like BeautifulSoup to scrape websites, filter the data appropriately, and create Document objects to build a VectorStore. They note that this is a baseline approach and there are many ways to improve performance.

Another community member has tried this approach but is not getting good performance, so they are asking for a better way to get information from a website and respond, as they are building a chatbot to provide information about a company from the company's website.

A third community member suggests trying different approaches like SubQuestionQuerEngine, Reranking, and Finetuning to improve performance. They emphasize the importance of the data ingestion step and thinking about whether a human would be able to understand the information being processed. They also mention that a good system prompt can sometimes help.

There is no explicitly marked answer in the comments.