Object index

Yes and no.

You can persist and load it, using the underlying obj_index._index attribute

But you need to maintain the object to node mapping. Then you can re-construxt using the class init

Generally I don't think you'd have more than 30 items in there, it might not be worth the effort of saving it (embeddings are cheap anyways 😅)

Well it takes me 30+ seconds to create it

and I don't want to use 30+ seconds each time

so when you say reconstruct using init, how do I do that

It should just be created on app startup though? 🤷‍♂️

Dig into the source code a bit, you'll see what I mean haha

gotcha

understood

you're awesome as always, thanks!

one more question about indices in general - I have a folder "docs", of folders, I make indices for each subfolder, get query engines, make them tools, and put it in agents. I persist the indices for each folder under their equivalent sub folder inside a "storage" folder. On app startup, I loop over the directories and load_index_from_storage for each of those directories

I just saw that the source code also has load_indices_from_storage

Would that be a better/more efficient way to load indices all at once?

Hmm, not quite understanding index store and index structs

You can save all indexes into a single folder yes, but they need to share the same storage_context object when creating them

The index_struct basically just keeps track of which nodes ids are available to the index

Example here
https://discord.com/channels/1059199217496772688/1158792048309174384/1158792737466888242

Then when you load, specify the index ID of the index to load