How do I add system prompts for my

You can define system prompt inside the service context itself with llm and embed_model.

@WhiteFang_Jr Can I add document id while creating a documents and query based on the document id?

Yes you can add doc ID to document. By default each document gets unique ID. You can modify it with


To query it via docID, i would suggest you add the uniqueId to the metadata and then do metadata extraction with query:
https://docs.llamaindex.ai/en/stable/examples/metadata_extraction/MetadataExtractionSEC.html

I am alrready following the same approach:

documents = []
for doc in docs:
if "content_text" in doc:
doc_content = doc["content_text"]
doc_id = doc["document_id"]
print(doc_id)
documents.append(
Document(
text=doc_content,
metadata={"file_name": docid,"date": str(datetime.now())}, id=doc_id,
description=f"Useful to answer question related to file name or id {doc_id}",
)
)

But I am not getting correct answers

Give this article a try: https://docs.llamaindex.ai/en/stable/examples/metadata_extraction/MetadataExtractionSEC.html

a design choice question, why did you guys choose to put system prompts in service context. can the system message be changed at each llm call?

Yes @OceanLi

IMO, Almost all the new LLMs have system_prompt which is a different instruction that you add with your text along with the other instructions that you may or may not add along with the text being sent to LLM.
For instance in case of GPT3.5

• System prompt
• user text ( +-) any other instruction that you want to add [ Done via prompt template in llamaindex ]

So I think putting it service_context makes it easier for user to add/edit and be easily available everywhere

I can be wrong though 😅

Plus to answer your query for each llm call if done via .query() method it will remain the same unless you change itand then call the .query() again

Yeah the system prompt will be made dynamic

Can you help me with streaming response ?
I am getting an error:

for text in streaming_response.response_gen:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Response' object has no attribute 'response_gen'

This error is coming as the response that you have is not a streaming response

Yeah but I have kept the streaming true

response_synthesizer = get_response_synthesizer(
streaming=True,
response_mode="tree_summarize",
)

But the query where actual response is being generated is not supporting streaming

Ok

Any ideas on how I should go about this?

I think you'll need to add the streaming_query method as well in your custom class

You can check on this file to chekc what kind of response you are getting: https://github.com/run-llama/llama_index/blob/b8e565f24d2312d3c6ca8120c1b557e14c77e259/llama_index/response_synthesizers/base.py#L99
I guess this will help you to debug further

class RAGStringQueryEngine(CustomQueryEngine):
"""RAG String Query Engine."""

retriever: BaseRetriever
response_synthesizer: BaseSynthesizer
llm: OpenAI
qa_prompt: PromptTemplate

def custom_query(self, query_str: str):
nodes = self.retriever.retrieve(query_str)
context_str = "\n\n".join([n.node.get_content() for n in nodes])
query_str= qa_prompt.format(context_str=context_str, query_str=query_str)
return self.response_synthesizer.synthesize(query_str, nodes)

this tweak worked

Awesome!!