Not sure about elsewhere, but in the US, it is possible to spoof phone numbers. The only way to verify that the person you are talking to is actually associated with the number is to hang up and call them back. This is particularly a problem for companies like banks and insurance.
Android actually does provide some amount of filtering[1], but if someone, say, spoofs a seemingly legitimate number, it will come through fine.
[1]: https://www.androidpolice.com/2016/07/25/googles-phone-app-n...