At the moment, the percentage of spam calls that bother to leave a message is low enough that it seems to work out. I also don't have to worry about the impoliteness of hanging up when it's a live human on the other end, and can time-shift the annoyance to when I'm not in the middle of something.

I wish I could ban the hundreds of 2-4 second voicemails I get a week. actually ban the number too.

Hmmm, as interrupting to answer at the time has the additional overhead of task/context switching, I'd say it's more.

If you just switch to/from the voicemail context once for 10 messages, the cost is only 10% of the cost for real-time answering.