how many files can you index lets say I

how many files can you index? lets say I wanted to have an entire codebase as an index is that possible?

12 comments

Yes, it will be loaded into your memory during runtime. So directly depends on your memory size.

Plus same space required in your storage disk to keep the embeddings locally if you are using VectorStoreIndex

SShinzō Abe

what would be the best appraoch if lets say, I have a developement side of things and also financial marketing data. Would it be a bad idea to have a huge lump of mixed data so whenever someone queries they have access to everything? or split it off

SShinzō Abe

is the first approach even viable?

WWhiteFang_Jr

If the docs are different let say Marketing has nothing to do with the developmnt code then it would be better for you to have separate vectors.

I'm hoping that these two can be launched separately or you would have to set some conditons that user selects if he wants to ask query over dev docs or financial docs, Based on the choice simply pick the vector store and proceed accordingly.

SShinzō Abe

wait, how could I launch multiple vectors at once? I am hoping to do this but I am not sure how to do it :D

WWhiteFang_Jr

Yes you can, It will be something like this

Plain Text

dev_docs = SimpleDirectoryReader(DEV_DOCS_PATH).load_data()
financial_docs = SimpleDirectoryReader(FINANCIAL_DOCS_PATH).load_data()
index_dev_docs = GPTVectorStoreIndex.from_documents(dev_docs, service_context=service_context)
index_financial_docs = GPTVectorStoreIndex.from_documents(financial_docs, service_context=service_context)


docs_query_engine = index_dev_docs.as_query_engine()
financial_query_engine = index_financial_docs.as_query_engine()

# for dev query
response = docs_query_engine.query("what is the status on deployment")

# for finance
response = financial_query_engine.query("How to become financially independent")

Now you have two query engines for two different docs, Choose the one based on your condition

SShinzō Abe

No I meant it the other way round where, everything wound be under one index

SShinzō Abe

Would that be the preferred way?

WWhiteFang_Jr

There's no preferred way. It depends on your usecase, If all the docs are interconnected then it would be better to have them together. And if not then keeping them separately would be a good choice.

SShinzō Abe

ok cool, thank you. One more question. Is it possible to append to an exising vector db if new data emerges? more of, is it a simple or a tedius task?

WWhiteFang_Jr

Yes, its a simple task.

Plain Text

# existing_index
index_dev_docs = GPTVectorStoreIndex.from_documents(dev_docs, service_context=service_context)

# new data
new_docs = SimpleDirectoryReader(NEW_DOCS_PATH).load_data()

# Add to existing index 
for docs in new_docs:
    index_dev_docs.insert(docs)

SShinzō Abe

omg this is amazing ty so much

Add a reply

Find answers from the community

how many files can you index lets say I