You need to add instructions in the prompt about what the llm should follow or not. The more clear the instruction is , better chance for the llm to follow it.
Also it depends on the llm you are using. if it is open-source llm, following the instructions set in the prompt may not go your way always.