And for every person getting scammed by an app from any source, there are a thousand people getting scammed just through phonecalls. Scam apps isn't a real problem at scale, it's a bunch of fear mongering.
And two wrongs don't make a right
If we ban sideloaded apps under this reasoning, we have to ban the Play Store as well, ten times more urgently.