Good morning! I have a question about sharing a knowledge base among users. I'd like to create a solution that stores all data in a single vector store or database. However, I want to restrict user access to only certain portions of the data. Is there a way to namespace the knowledge base so that each user can only access their designated areas?
I have seen comments from this community that a list index with several smaller indexes may be better than one huge index. I have not tested this thoroughly myself yet, but I am currently using list indexes to combine vectorized datasets, and this approach has some advantages. One of the big ones for me is, how to you get stale data out of a huge index? That is a hard problem, but if you have a number of smaller indexes, you can reindex individual parts as needed and easily discard the old data. This method may also be the best way to handle any sort of role based controls. You can vectorize different portions of your dataset into distinct files, and then create different combinations available through different list index, like list index 1 has data sources a, b, c & d. List index 2 has only a & b, list index 3 has only c &d, and you can create a list index with only the relevant datasets for a given role/permission scope.
