Find answers from the community

Updated 3 months ago

Non-OpenAI models not respecting system prompts/not consistently retrieving

Hey, y'all! I have an app set up where users can switch between different a swath of different settings. One of those settings switches out the ChatEngineMode between Context, Best, FLARE, HyDE, and the BaseQueryEngine. I want every query to perform a retrieval, but am seeing the engine spit out a pure LLM response a good amout of the time, despite my efforts to tweak the system prompt and ReActChatFormatter. Is there a way to force the llama_index engines to retrieve before responding? They don't seem to be respecting the system prompt at all.
H
L
4 comments
The issue is pretty much exclusive to non-OpenAI models. This is the code I'm running the other models (Mixtral-8x7b/22b, Llama3-8b/70b-8192, Gemini Pro/1.5 Pro, all three Claude models, Command-R) through:

Plain Text
from llama_index.core.agent.react.formatter import (
                        ReActChatFormatter,
                    )
                    from llama_index.core.agent.react.prompts import (
                        REACT_CHAT_SYSTEM_HEADER,
                    )

                    fsh_format_base = "{prompt}\n{system_header}"
                    final_system_header = (
                        REACT_CHAT_SYSTEM_HEADER
                        if self.get_setting("system_prompt") is None
                        else fsh_format_base.format(
                            prompt=self.get_setting("system_prompt"),
                            system_header=REACT_CHAT_SYSTEM_HEADER,
                        )
                    )

                    react_chat_formatter = ReActChatFormatter(
                        system_header=final_system_header
                    )

                    chat_engine = self.index.as_chat_engine(
                        chat_mode=chat_engine_type_lower,
                        llm=self.llm,
                        query_engine=query_engine,
                        memory=memory,
                        system_prompt=self.get_setting("system_prompt"),
                        streaming=self.get_setting("streaming"),
                        similarity_top_k=self.get_setting("similarity_top_k"),
                        react_chat_formatter=react_chat_formatter,
                        context_template=context_template
                    )
Non-OpenAI models not respecting system prompts/not consistently retrieving
not really a way to force the react agent, unless you implement a custom ReActAgentWorker
Gotcha. I'll look into that, then
Add a reply
Sign up and join the conversation on Discord