You might like the book Snow Crash. Spoilers: One of the plot points is that looking at certain visual pattern (in the form of tv static/snow) can permanently destroy someone's mind. The later parts of the book explore auditory signals.

This premise is somewhat believable since there are already people affected by epilepsy and can't tolerate flashing lights. And even regular folks are susceptible to optical illusions with (short-term)lasting effects such as https://en.wikipedia.org/wiki/McCollough_effect, https://gfycat.com/CoordinatedSleepyAnole, etc.

I wonder if they are really some combination of sensory input that can like you say "drive oneself mad" after exposure.

Side note: Optical illusions also affect artificial neural networks where they misclassify noise with high confidence as real objects: http://karpathy.github.io/2015/03/30/breaking-convnets/