For a clear example, imagine that I'm roaming in Zimbabwe with my UK cellphone. I send an SMS through the Zimbabwe carrier. It (eventually) arrives to the UK recipient network, ready to be delivered. That network could do some form of verification, but as they only get the final billing tally a few days or weeks later from the Zimbabwe ISP, they don't have enough information to do so.
It would not make any sense for the carriers to do SMS verification. And given that emails are far easier to get people to click on links to phishing and malware sites, spoofing SMSs has limited value.
Also, did you know that I could phone you and claim to be someone else?