[Question]: How do I get the exact and f...

ssiarez

Hi, I have a question that went over discords character limit, so I created an issue here: https://github.com/jerryjliu/llama_index/issues/7750
Any pointer are appreciated.

8 comments

LLogan M

try this instead
https://gpt-index.readthedocs.io/en/stable/end_to_end_tutorials/one_click_observability.html#simple-llm-inputs-outputs

LLogan M

I recently fixed some stuff surrounding the query-wrapper-prompt. Make sure you have the latest llama-index version 🙂

ssiarez

Thanks, I upgrade to the latest, and called set_global_handler("simple")
It still looks like query_wrapper_prompt is not used.

LLogan M

Can you share more of your code? I had this working for llama2 just the other day

ssiarez

Funny enough, now as part of the " Completion ", it prints out my query_wrapper_prompt even though it seems to be missing from the ** Prompt: **
So I'm wondering how it knows about my query_wrapper_prompt if it is missing from the prompt.

I can share my code. It's one of the notebook examples that I made changes to. (I won't be able to share the source document that I'm using to create the index though)

LLogan M

hmm super strange 🤷‍♂️ the set_global_handler("simple") Should be printing the exact input / output to the LLM

The LLMPredictor is what adds the query wrapper to the input, and then it calls the llm from there

Plain Text

formatted_prompt = prompt.format(llm=self._llm, **prompt_args)
formatted_prompt = self._extend_prompt(formatted_prompt)
response = self._llm.complete(formatted_prompt)

LLogan M

(extend prompt is what is adding the query wrapper)

LLogan M

Just for fun, I confirmed the behaviour works fine on my end 🤔

Plain Text

(llama-index) loganm@gamingpc:~/llama_index_proper/llama_index$ python
Python 3.11.0 (main, Mar  1 2023, 18:26:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from llama_index import LLMPredictor, ServiceContext
>>> from llama_index.prompts import Prompt
>>> from llama_index import set_global_handler
>>> set_global_handler("simple")
>>> from llama_index.llms import OpenAI
>>> llm = OpenAI(model="gpt-3.5-turbo-instruct")
>>> ctx = ServiceContext.from_defaults(llm=llm, query_wrapper_prompt=Prompt("[INST] {query_str} [/INST] "))
>>> response = ctx.llm_predictor.predict(Prompt("Hello world"))
** Prompt: **
[INST] Hello world [/INST] 
** Completion: **


Hello world


>>> response
'\n\nHello world'

I'm using openai here, but the LLM itself doesn't matter. I did remove the extra **** characters from the code though, they seem to cause print buffer issues or something

Add a reply

Find answers from the community

[Question]: How do I get the exact and f...