Llama Index Setup for Knowledge Graph and LLM Agents

At a glance

The community member is evaluating the LlamaIndex library to build a knowledge graph or long-term memory for their LLM agent. They have a few questions, including whether there is an S3 bucket loader in LlamaIndex, and why they need to pass the vector store when creating the storage context, when the storage context itself knows the vector store being used.

In the comments, another community member provides some helpful information. They confirm that there is an S3 bucket loader in LlamaIndex, and explain that if you are creating the storage context, you need to pass in the vector store reference, otherwise it will create a default one. They also mention that if the vector store service (like Chroma or Qdrant) goes down, you may need to re-do the embedding process to recreate the embeddings.

Another community member asks for more context on the storage context and when to use it, as well as why they might need to store documents and the index separately. The first community member suggests referring to the LlamaIndex documentation for more information on the storage context.

Useful resources

ppurple crow

Hello just starting to build a knowledge graph or long term memory for my LLM agents

evaluating llama index for the same.

Couple of quick questions -

is there a s3 bucket loder in llama index
i dont understand vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
why do i need to send the vector store while load index and storage context both when storage context itself knows whats the vector store used ?

do i need to store the embeddings at some blob storage as well, incase my vector store goes down i might need to recreate all of those ?

3 comments

WWhiteFang_Jr

Hey!

Yes there is: https://llamahub.ai/l/readers/llama-index-readers-s3?from=
How are you loading back the index,

Plain Text

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

If you are creating storage_context then you will have to pass in your vector_store ref else it will create a default one on its own.

You can, if the service goes down like ( Qdrant or chroma, which has never happened to me ) , then yeah if the service does not gets up correctly you'll have to re-do the embeddings process again.

ppurple crow

thanks @WhiteFang_Jr for answering

if u have more context on the storage context, can u tell me more on when to use it as why would i need to store documents separately, index store separately etc ?

WWhiteFang_Jr

yeah sure, you can read more about storage context here: https://docs.llamaindex.ai/en/stable/module_guides/storing/

Add a reply

Find answers from the community

Llama Index Setup for Knowledge Graph and LLM Agents