Another question - I used azure openapi's capability to build a chatbot over your data to see how it compared.. Lots of thoughts I can share there ... But one neat feature was that it included the source document and position within the source document as references. I want to reverse engineer this into my engine - how do I go about doing this ?
I think how they are doing this is using openai's new function calling API, and creating pydantic objects that let the model explicitly state a source.
Now, there's chance it might hallucinate this lol but it seems to be fairly reliable. Been thinking a bit about adding this to llama-index but I haven't had the time
I plan to do a direct comparison of the two. It uses azure cognitive services to build up an index and then uses that. In my experience so far it takes much longer for Azure to build the index, there are really low max quotas out of the box and it could get expensive quickly. The actual responses... I created a basic vector index in llama and threw the same data at it as I did in Azure. Response times were pretty similar as were the responses.
That is a massive pat on the back to llama index - if I were to put a few more hours in, tune the parsing, the prompts etc it would be more accurate, faster and loads cheaper
For the citations, we do have something slightly similar. It was built before the function calling api came out, so it could probably be revamped a bit at some point
It basically takes any index and breaks the nodes into slightly smaller numbered nodes, so that the LLM can cite things properly
