This is a dumb question but is the value prop of llama index in that I can index my entire dataset, and behind the scenes, it goes through all of it in chunks querying GPT? So, for instance, I have a bunch of discord chat logs. I want to be able to query GPT to identify and let me know all questions that were asked in my indexed logs. I can do this by looping through the logs and passing them in as chunks to GPT. the advantage of llama-index is that it would do this behind the scenes for me, so I don't have to handle the looping?
Correct! but also, Llama Index provides a bunch more.
Different types of indexes (list, vector, knowledge graph, tree, keyword), building indexes on top of eachother (i.e. vector index on top of a list index), integrations with 3rd party vector stores, and more!
For your use-case of discord logs, I would definitely recommend llama index over doing something manually
Very cool. I'm guessing a simple list index will work here, so it's just an array of chat logs. GPT can easily identify questions, I just need a way to have it return some ID of a chat log to let me know which ones it identified as questions. I think I can do that by dumping my documents and prefixing each log with a unique identifier and in the prompt ask gpt to return said ID
