Find answers from the community

Updated 3 months ago

[Question]: How to print the final promp...

At a glance

A community member is trying to get the final processed prompt being sent to a Large Language Model (LLM). They found a related post answered by @Logan M, which provided some code to handle LLM events. The community member is confused about how to incorporate this code into their own code, which uses a vector store index and a query engine.

The comments suggest that the community member should run the dispatcher code at the top before creating the query engine/index. However, when the community member asks questions to the query engine, they only see the response without any logging. They ask if they need to initiate logging or something to see the prompt sent to the LLM.

After some back-and-forth, the community member discovers that they need to remove the "name" parameter from the dispatcher code and add "event: BaseEvent" as an input to the "handle()" function in the "ExampleEventHandler" class. They also add some "flush=True" statements, and this seems to make the logging work as expected.

Useful resources
Hi, I'm trying to get the final processed prompt being sent to LLM. I searched online and noticed a related post answered by @Logan M : https://github.com/run-llama/llama_index/issues/13310
The code provided is:
from typing import Dict, List


from llama_index.core.instrumentation.events.llm import (
LLMChatEndEvent,
LLMChatStartEvent,
LLMChatInProgressEvent,
)


class ExampleEventHandler(BaseEventHandler):
events: List[BaseEvent] = []

@classmethod
def classname(cls) -> str: """Class name.""" return "ExampleEventHandler" def handle(self) -> None: """Logic for handling event.""" print("-----------------------") # all events have these attributes print(event.id)
print(event.timestamp)
print(event.span_id)

# event specific attributes
if isinstance(event, LLMChatStartEvent):
# initial
print(event.messages)
print(event.additional_kwargs)
print(event.model_dict)
elif isinstance(event, LLMChatInProgressEvent):
# streaming
print(event.response.delta)
elif isinstance(event, LLMChatEndEvent):
# final response
print(event.response)

self.events.append(event)
print("-----------------------")


import llama_index.core.instrumentation as instrument

dispatcher = instrument.get_dispatcher(name)
dispatcher.add_event_handler(ExampleEventHandler())


However, I'm confused about how to incorporate this code into mine. Basically I have a vector store index built from nodes, and I'm using it as query engine to ask questions.

from llama_index.core import VectorStoreIndex

recursive_index = VectorStoreIndex(nodes=base_nodes + objects)
recursive_query_engine = recursive_index.as_query_engine(
similarity_top_k=5,
verbose=True,
response_mode="compact"
)

Thanks for your help!
L
l
11 comments
That code already plugs directly into yours?
Just run that dispatcher code at the top before creating your query engine/index
Don't know if I miss anything, but when I ask questions to the query engine, it only shows the response without any logging. Here's my code.
do I need to initiate logging or something to see the prompt sent to LLM? Sorry if I asked dummy questions, and I really appreciate your help.
Hmm, I made this change

Plain Text
dispatcher = instrument.get_dispatcher()
dispatcher.add_event_handler(ExampleEventHandler())


And that was the change that made it work
seems like you remove the name inside the parentheses. I tried to do the same, but there's no logging of LLM prompt during query-asking.
Note that the code has a print_source_nodes(response) function, which prints out the texts of the top 5 retrieved nodes. However, the printed results are not the final prompt sent to LLM.
you may comment out the last line: print_source_nodes(res1). Now you may only see the logging from asking the query.
Works fine for me on Google colab
ah, looks like your handle() function under the self defined ExampleEventHandler class is different from mine. My code forgets to include event: BaseEvent as an input. I also added a bunch of "flush=True" as shown in your code. Now looks like it's working! Thank you so much for your help!
Add a reply
Sign up and join the conversation on Discord