My fear would be something like a friend or family member being in some emergency situation where they have lost their phone and need to reach me on someone else's phone but I never see the call/text. There's a danger in micro-managing your life without fully thinking about all the ramifications. This is why in my opinion it's better to just deal with the 1-2 seconds of annoyance to dismiss a spam call/text. It's not the end of the world. A child/SO/friend/family/co-worker not being able to reach you in an emergency is much worse.