is there better control for indexing

Can you give an example of what you mean? 🤔

I am trying to load a DnD Module, into a model to Be a DM. Modules have all the piece contained to play a full game, all the bad guys, weapons, rules a story arc and MAPS. My issue is the rooms for the map were numbered, room 8 is the Great Hall. After indexing Room 8 is now a the armory with 4 guards. Every thing about the armory is correct but it no longer matches the player hand out.

ah I see. Is everything being loaded from a single dcoument file?

I set it up 2 times. Once as compiled indices with 7 chapters as tree indexes with a keyword graph on top like the demo. The other had the entire manual as a simple vector index from the langchain demo. Both methods had errors.

Todays approach was to try to load a 30 page promp so no data would be lost from indexing. Im just trying to figure out how to load .txt into the model.

Ah I see! The tree index depends entirely on the LLM to summarize things correctly. I see it's probably missing some information sadly.. :PSadge:

I would recommend using a vector index for each chapter maybe, then a keyword on top?

Will i get a better result if I break it up by sub chapter?

Maybe! I would start with chapters first and see how well it works 🤔

What Vector index do you recommend

If it's not too much text, a simple vector index should be fine.

If you notice it's using a lot of RAM, maybe switch to using something like pinecone or qdrant

Apologies for the noob question - but how exactly does one tie keywords to specific indexes? Is there some documentation I can look at, as I am not able to find this anywhere...

You might be interested in this: https://gpt-index.readthedocs.io/en/latest/guides/usage_pattern.html#setting-required-keywords-and-exclude-keywords

Thanks a million!I see it's an option in querying, but am unsure how you use that against a vector store such as pinecone, where only the vectors are stored? Do you know of vector store where this is a key functionality?

I thiiiink it should still work with pinecone? Maybe give it a shot.

Basically it's filtering nodes based on the text in the node. So it should still have access to that with any vector store right?

You're probably right! Will test for sure. Thanks a million!