That’s not really a solution to this either, because security can’t just be for people who know how to compile and side load apps.
Of course, there's still plenty of possible supply-chain attacks and the like. Closed source app store, git repositories taken over or sneaking in malicious commits, binary blobs required to run your device, and so on. But we should encourage any progression in security, and turn good security practices into common sense.
I’m not arguing that you shouldn’t allow side loading.
But let’s say this is what developers start doing. We compile our own code and side load. Great for us. How many people can each individual support? Immediate family and close friends? Are most people on their own? Now say your immediate contacts are compromised. That exposes some of your details as well would it not? At least you’d be more vulnerable to social engineering.
So maybe you have to work towards a system where everyone is safe.