In your case, a dedicated attacker could socially engineer the carrier into changing you and your father's contact details and then port the number.
$100k is a lifetime's worth of money in some countries, and it can justify a few months worth of recon.