I guess I don't think about that problem because all of the buttons and switches in my life already solve that issue by being made of plastic or metal.
That also just reminded me of another big downside to wood: all the dogs I've ever owned have been really interested in chewing on wooden objects left around the house, even if they were otherwise perfectly respectful of everything else. I've lost quite a few shoe-polish brushes over the years.