Llm predictor

At a glance

The community member was trying to change the llm_predictor when creating an index file. They initially used text-ada-001 as the language model, but other community members suggested using gpt-3.5-turbo instead, as it is 10x cheaper than the default model. The discussion also covered the chunk_size_limit parameter and how it affects the index creation and query process.

The community members also discussed the use of LangChain and Llama Index for building a customer service bot, with the recommendation to use LangChain agents with Llama Index as a custom tool, rather than relying on fine-tuning, which can be difficult and lead to unexpected results.

Finally, the community member encountered an issue with installing the required packages, specifically llama_index, and the other community members suggested installing the package again and checking the latest documentation for any changes in the interfaces.

Useful resources

OOri

I was trying changing the llm_predictor when creating the index file : def construct_index(directory_path):
max_input_size = 4096
num_outputs = 512
max_chunk_overlap = 20
chunk_size_limit = 600

prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

llm_predictor = LLMPredictor(llm=OpenAI(temperature=0.7, model_name="text-ada-001", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

index = GPTSimpleVectorIndex(documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper)

index.save_to_disk('index.json')

27 comments

LLogan M

Is text-ada-001 a valid LLM?

Try something like gpt-3.5-turbo using the ChatOpenAI class, it will be 10x cheaper than the default LLM

OOri

Do you mean "text-davinci-003" will be 10x cheaper than the default LLM ? (https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_llms.html)

LLogan M

No, text-davinci-003 is the default, and it is $0.02/1k tokens

gpt-3.5-turbo is $0.002/1k tokens

OOri

Like this ?: llm_predictor = LLMPredictor(llm=OpenAI(temperature=0.7, model_name="gpt-3.5-turbo", max_tokens=num_outputs))

OOri

cause no mention to text-davinci-003 : https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_llms.html

LLogan M

Try something like this

Plain Text

from llama_index import LLMPredictor, ServiceContext
from langchain.chat_models import ChatOpenAI

llm_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo"))
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, chunk_size_limit=1024)

LLogan M

Chunk size limit is optional haha

OOri

first thanks 👍
I would appreciate a brief explanation.
And what is Langchain used for here ?

LLogan M

Langchain is just used as a wrapper around the api calls to openAI. Llama index uses langchain for a few small things to avoid duplicating code

LLogan M

So they have the specific wrapper to call gpt-3.5 already written

LLogan M

Then it gets wrapped with the llama index specific object

OOri

Hi Can you explain the role of this line : service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, chunk_size_limit=1024 ? is it mandatory ? what do i need to do with service_context?

LLogan M

The service context sets up certain parameters about how the index will do things. This includes the llm_prediction, embedding model, node parser, prompt helper, chunk size limit, and llama logger.

By using from_defaults, it sets everything to the defaults, except for what we pass in. Then, you can pass it into the index constructor or query calls

index.query(..., service_context=service_context)

index = GPTSimpleVectorIndex.from_documents(..., service_context=service_context)

It's only needed for when you want to change one of the settings that the service context holds. In my example, I changed the llm_predictor and the chunk_size_limit

OOri

Cannot even load the llama_index.... from llama_index import LLMPredictor, ServiceContext
ModuleNotFoundError: No module named 'llama_index'

OOri

Cannot even load the llama_index.... from llama_index import LLMPredictor, ServiceContext
ModuleNotFoundError: No module named 'llama_index'

LLogan M

sounds like something broke in your env 😅 maybe try installing again? pip install --upgrade llama_index

OOri

WOW Logan, You are the champ!!

OOri

What is the default for : chunk_size_limit ? Does this parameter affect the length of the answer in the query? Or how to create the index file?

LLogan M

It changes how your documents are chunked

If an input document is greater than that token length, it gets broken into chunks that are up to that size

I find 1024 works best for emebddings (and in newer versions, will actually be the default). The current default is 3900

By default, the vector index fetches the top 1 closest chunk that matches the query. However, since we are lowering the size, you might want to fetch more chunks

res = index.query(..., similarity_top_k=3, response_mode="compact")

The response mode there just stuffs as much text as possible into each LLM call, rather than making one LLM call per chunk/node. Will use the same number of tokens, but hopefully be a little faster

OOri

What is your recommendation for a customer service bot for a saas platform, should I use LLamaindex or fine-tuning process ?

LLogan M

tbh fine-tuning is pretty difficult, it can lead to some pretty unexpected results.

I would use something like a langchain agent + llamaIndex as a custom tool for the agent. You can customize the prompts for the agent to hopefully behave the way you want it to.

Additionally, you might need some sort of extra filtering layer to ignore bad inputs or outputs. One approach I've seen is to setup a large list of expected inputs, and if the similarity between that items from that list and the users text is too low, refuse to answer lol

LLogan M

Fine-tuning is usually a last resort, at least from what I've seen

OOri

Any idea how to implement chat history with LLM ?

LLogan M

Yea, for that you'd probably want to use langchain+Llama Index as a custom tool

Llama index on its own is more of a search engine

OOri

Hey Logan you are the only one that can help me.... Unfortunately I had to reinstall all packages python from scratch on a new computer and now I have this error which I do not know how to struggle with it : Python 3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.

= RESTART: C:\Users\orish\Desktop\PythonProjects\Fine tune bot - Read doc folder files\app - simple 3.5 chat.py
C:\Users\orish\AppData\Local\Programs\Python\Python311\pythonw.exe
Traceback (most recent call last):
File "C:\Users\orish\Desktop\PythonProjects\Fine tune bot - Read doc folder files\app - simple 3.5 chat.py", line 4, in <module>
import gpt_index
ModuleNotFoundError: No module named 'gpt_index'

OOri

I have tried to install this package few times to update it I don't know what is the problem

LLogan M

You'll want to install and use llama-index, instead of gpt-index

Just a heads up, a lot of interfaces also changed recently. Check out the latest docs for a refresh
https://gpt-index.readthedocs.io/en/stable/guides/primer/usage_pattern.html

Add a reply

Find answers from the community

Llm predictor