is it possible to not download model llama-2-13b-chat.Q4_0.gguf if I already have one?



If I select llm_predictor to the same LLM, im getting error:

Error 'list' object has no attribute 'encode'
on this
response = chat_engine.chat("What is this book about?")

my code is from tutorial https://gpt-index.readthedocs.io/en/latest/examples/chat_engine/chat_engine_react.html#get-started-in-5-lines-of-code

can i specify additional metadata to messages when doing chat_engine.chat("mymessage") ? for example user name, chat room name, date and time

If Im right, chan_engine automatically update index with new messages and store it if persist specified.
Or maybe I should somehow chat_engine.index.update() and refresh() ?

If I load index from storage for faster startup, then how to update that index if there is new documents in folder? I need to load documents everytime and check is there new docs and need to refresh index?

The OpenOrca-Platypus2-13B model supports this template for chat:


But llamaindex react loop agent dont use this format. there is many \n and examples like '''Please use valid JSON.......
Observation: tool response....
...

Can it be really difficult for custom LLM?How to fix?

How to use with Jupyter Notebook? Getting error trying to load LlamaCPP

UnsupportedOperation Traceback (most recent call last)
c:\LLMs\SharikAI\My_llama_index.py in line 5
33 callback_manager = CallbackManager([llama_debug])
34 context_window=3900
----> 35 llm = LlamaCPP(
36 model_path='C:/LLMs/oobabooga_windows/text-generation-webui/models/openorca-platypus2-13b.Q4_K_M.gguf',
37 temperature=0.3,
38 max_new_tokens=512,
39 context_window=context_window,
40 generate_kwargs={"top_p": 1,},
41 model_kwargs={"n_gpu_layers": 32, "n_batch": 512, "n_threads": 10},
42 # callback_manager = callback_manager,
43 verbose=False,
44 messages_to_prompt=messages_to_prompt, # The function to convert messages to a prompt
45 completion_to_prompt=completion_to_prompt, # The function to convert a completion to a prompt.
46 )

File c:\LLMs\SharikAI.venv\Lib\site-packages\llama_index\llms\llama_cpp.py:110, in LlamaCPP.init(self, model_url, model_path, temperature, max_new_tokens, context_window, messages_to_prompt, completion_to_prompt, callback_manager, generate_kwargs, model_kwargs, verbose)
105 raise ValueError(
106 "Provided model path does not exist. "
107 "Please check the path or provide a model_url to download."
108 )
109 else:
--> 110 self._model = Llama(model_path=model_path, **model_kwargs)
...
370 else:
371 msg = "fileno"
--> 372 raise io.UnsupportedOperation(msg)

UnsupportedOperation: fileno

Why when using index.as_chat_engine(..., chat_mode=ChatMode.BEST ) it wont go in react loop, but gives only one short answer?

How to merge book index and chat together? So I can ask LLM about book in chat manner?

What if LLM dont want to use query_engine_tool and give me answers about some imaginary book, not I embedded to it?


Response: Observation: query_engine_tool response
Title: The Da Vinci Code (WRONG!!!)
Summary: This book is a thriller that follows Robert Langdon, a Harvard symbologist, as he unravels ancient secrets and solves codes to save the life of a British Royal Family member.

Main characters include:

1. Robert Langdon - A Harvard professor of symbology.

....blabla


my code


Directory contains 1 big docx file converted from fb2 and translated to English with google