The community members discussed the scope of the index, with one member suggesting that there is no limit to what can be added to an index. However, they noted that with a large number of documents, it may be more efficient to use a vector store like Pinecone or Qdrant. Some community members suggested organizing documents into topics and creating an index per topic, which could then be used as tools in a Langchain agent or wrapped with a top-level vector or keyword index to help route queries to the right data. The discussion also touched on the idea of building a massive graph data structure and the use of vector stores.
No limit! You can add as much as you want to an index. But with that many documents, you might want to look into using a vector store like pinecone or qdrant to be more efficient
Some people have had luck organizing documents into topics, and then creating an index per topic.
Then, each index could be a tool in a langchain agent, or you can wrap them all with a top level vector or keyword index to help route queries to the right data
