One solution, not practical for most folks but works well for me: live in a different area code than phone number. (My cell number is in the area code I grew up in.)

It seems most spam calls either fake a source number that's in my phone's area code, or use a completely random source. Almost any inbound call I actually expect without an address-book entry would come from my (new) local area code, so I can pick up a call if (in current area code) OR (in address book). So in essence, my current area code is a 3-digit passcode.

Perhaps a nice trick if one is allocating a new VoIP number -- at the cost of looking like a non-local when giving it out, you can effectively screen for actual local calls...