LlamaIndex is great Although have some

PPunkbit

LlamaIndex is great! Although have some thoughts.

There are a lot of services online that provide a ChatGPT like experience for own documents, and I notice that all provide a different answer based on the same documentation sources. As most of this services seem to copy each other, and the pricing, thought they're using LlamaIndex or similar but yesterday, when indexed a bunch of markdown files noticed that LlamaIndex + GPT3 responds with what seem to be poorer answers. For example, my markdown has text description and source-code like this

Plain Text

cmd arg1 arg2

👆 Which is included in the answer, but I don't seem to get the same result with LlamaIndex.

How can I improve my LlamaIndex to be closer to the answer I get with third party services?

16 comments

LLogan M

If you are using a vector index, you might want to inspect which nodes were used to make the answers. If you notice that the answer isn't in the source nodes, try increasing similarity_top_k in the query (if you are using a vector index).

Plain Text

response = index.query(...)
print(response.source_nodes)

You could also play around with pre-splitting your documents into sections, rather than using the raw output from the loader.

Final option is looking into some prompt customization: https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_prompts.html

default prompts: https://github.com/jerryjliu/llama_index/blob/main/gpt_index/prompts/default_prompts.py

chat-specific prompts (for chatgpt and gpt4): https://github.com/jerryjliu/llama_index/blob/main/gpt_index/prompts/chat_prompts.py

Specifically, you might want to create custom QA and Refine prompts that have more specific instructions. You can pass in the prompts at query time index.query(..., text_qa_template=my_qa_template, refine_template=my_refine_template)

PPunkbit

@Logan M thanks a lot for your effort! I do the same and know how difficult it is. I work as a DX at Fleek, and found your project interesting for our https://docs.fleek.network

Alternatively, there are some SaaS which I tested that work fine, but given a shot to Llama 🙂 I'll take some time to go through your suggestions. Thank you very much!

cconic

@Punkbit What third party alternatives have you tried?

cconic

I'm running into similar issues with the results being weird. I'd like at least one paid for alternative.

PPunkbit

@conic I'm not sure if its the best thing ever but https://docsbot.ai/ I'll test the cheapest for a month

PPunkbit

@conic there are so many of this SaaS and most have a similar price, I find very suspicious, as they all might be using the same source, but after testing noticed different results and this one hold good results at least for the queries I made, I do believe that there's no perfect solution and nothing substitutes a human, so I'll do a test for a few weeks

PPunkbit

I do find Llama quite capable and refining it to our needs will be a better option in the long term

LLogan M

@Punkbit @conic if you have specific needs that llama index isnt providing yet, or general feedback/questions, feel free to reach out to the Google form in the enterprise channel!

We are trying to gather some feedback from users who are integrating llama into production

PPunkbit

@Logan M in my opinion, your team is offering this and seems so accessible that I don't really have anything to add. I do think that a different approach in the documentation can help onboard more users. I'm a soft dev > 15+, quite rare I pick python, but recognised the syntax, but there were a few things that I wasn't expected or not familiar with. I think with that out of the way, while onboarding someone into setting up the expectations for a running python environment will allow more users to learn it.

PPunkbit

With python dev env out of the way, its quite simple to have something running with just a few lines of code

PPunkbit

A few things I did was, to figure out how to add the required packages in a new project etc, so found about requirments.txt, etc.

PPunkbit

Created an http server with fastapi, as making http requests seem to be more sound to my needs

PPunkbit

Noticed that the text I got in the result, was trimmed, didn't complete the sentence.

LLogan M

@Punkbit awesome, thanks! I agree the docs need some work. It's been hard to keep up since the codebase is also evolving rather quickly lol

Did you figure out how to fix the truncated responses? OpenAI limits it to 256 tokens by default, but this can be changed. Check out this FAQ: https://discord.com/channels/1059199217496772688/1059200010622873741/1088122994251010139

PPunkbit

@Logan M oh gosh, if that was the problem, my bad! So, not really much to add apart from those comments regarding the documentation (not that is bad, but if it considers any person who haven't setup python etc might be interesting to take that into consideration, maybe it does and I didn't notice, not sure now).

cconic

Yeah sure perhaps we can start a thread somewhere about the thing I'm running into. I'll get to a computer in a few minutes

Add a reply

Find answers from the community

LlamaIndex is great Although have some