Pixel phones are now preventing sim swapping, IIRC, and maybe a few others.
I would understand Fi having better protection because you have to manage it online through your Google account (and same for Google Voice users).
Not sure what a phone would be able to do about it.
Anyway, Google as your phone provider opens up a different can of worms. Instead of having an overly helpful support staff who can be tricked into doing things they shouldn't, now you have Google's algorithmic account bans and tie-in with Google Payments and no recourse for problems because the customer service is a robot that says "no". https://news.ycombinator.com/item?id=18886804
Nonexistent support is great in terms of resistance to social engineering, but sometimes you need support.