Snooze alerts? Why don't you direct people to use team channels.. that way over time the knowledge base is there.
Also, nobody has to be put on the spot. Leaving them waiting a couple of hours max for an answer is usually the same wait time anyway. In the meantime you can also ask refining questions before producing the final answer.
Seriously. Keeping these conversations in DMs keeps the knowledge siloed and ensures the current situation will never get better.