Find answers from the community

Updated last year

Node mappings

Hey hey folks!. I hope you guys are having a fantastic week and I wish you a fantastic weekend ahead. I have a question: I am creating a list of nodes, and retrieving its mappings, similarly to how it is made in this (create_llama) example [https://github.com/run-llama/create_llama_projects/blob/5c136d5b561f7e2b806ecce9d802bdcf9f8d9c80/embedded-tables/backend/app/utils/index.py#L138] . The nodes are already stored in OpenSearch. Now, the question is: how to store the node_mappings effectively? in the example the mappings are pickle files, but those do not scale easily when loading thousands of mappings for thousands of files. Is there native support for a database to store them?, or does someone have an example of them stored in a scalable solution? (redis, opensearch, pg, s3). Thank you so much to everyone for the help and support.
L
T
3 comments
I would just store the node mappings in redis or some sql db yourself. Not too hard to do πŸ‘
you are right, but there might be other questions. For example, if I have thousands of files, loading the mappings in-memory might be expensive. The node_dict attribute of the RecursiveRetriever should be able to query them dynamically on a storage context built for that purpose. Is this a right way to see the challenge?
I can probably build a callback, but I just want to be sure that I am framing the problem correctly
Add a reply
Sign up and join the conversation on Discord