Hey! What would you recommend as best way to index a website?
I am using bs4 to crawl and format documents to SimpleVectorStore and chatGPT for llm_predictor but the results are sub optimal for bigger websites.
-> Often the answer is not found from context that chatGPT receives.