Are you sure that this is the case? According to https://support.google.com/fi/thread/761170 there was at least one case where support agents moved a Fi number from a locked account to a newly reopened account, without the user being able to access the locked account:
> I call Google Support and the amazing Google Support rep I spoke to (The only good one I’ve talked to so far during this whole process) stayed with me for over an hour and worked with tier 2 support in getting my Project Fi number transferred to Account 2.
A SIM swap with Fi might be harder to accomplish than with a regular provider, but it looks like there's still the risk that someone manages to convince a support agent to move the number to a different account.