Even if I know where the call is coming from, your experience shows it is still possible to mess up because you can't possibly account for all the kinds of weird phone systems people might be using.
But otherwise, I leave my phone on silent and have it set not to notify me about voicemails. My friends and family know how to reach me.
The good news I guess is that, while I'm sure people vary, spam phone calls and texts seem to have gone down quite a bit for me to the point where they're a very occasional annoyance.
The solution to me seems clear: use video/voip apps for everything and ditch the phone number. I don't know how to reach that critical mass. It feels like we should already be there with zoom and teams, but apparently not.