This is generally pretty easy, with ChatGPT at least, in my experience.
Langchain uses this prompt: "The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know."
Also, just don't ask it for things it couldn't possibly know without making it up like medical citations.
This is more for where you want your "AI" assistant to say it does not know how to book a reservation rather than pretend that it can.