Apple allows sideloading apps so that devs can work without the whole Apple deployment shenanigans. You write your app, sign it in Xcode, and load it onto your device.
This is allowed and encouraged.
F.lux did this to sign a compiled binary for their app. All good! Totally fine. The problem is that that they then distributed that compiled binary to people to sideload.
You're allowed to distribute source code and let people compile and sign it on their own Xcode – you are not allowed to distribute compiled code to sideload onto people's devices.
There are open source alternatives that now both predate and outlive flux on iOS, such as Gammathingy.
Apple decided to butt into this simple transaction and shut it down, despite the fact that they are not involved. But that's not so much "not allowed" as it is "displeases a big company that throws their weight around a bit too much."
If the app is open source, you're free to compile it with Xcode and install it on your device. That's fine, because the open source nature means you can see everything the app is doing and verify that it isn't malware (and if not you personally, then someone can do it).
But for closed source distribution, barring enterprise distribution, it has to go through the App Store. Allowing anything else opens up Apple's customer base to malware.
Closed source isn't a barrier to reverse engineering in any practical sense anymore. It's a post-IDA world.
Closed source is very much a huge barrier in verifying what software is doing, just as much as it always has been. I say that as someone who has been reversing engineering professionally for much of that time.
The number of people with the expertise and access to IDA is a tiny subset of those who can just skim source code. And those who are competent reverse engineers take 10x-100x longer going that route. An even smaller subset of those have the inclination to even bother doing this for free in their spare time.
Private APIs are a security vulnerability? Come on, you know better than that. It's not like a malware author can't take five minutes to come up with a way to bypass the private API checker they run on App Store submissions. The prohibition on private API is purely about not breaking apps when Apple releases OS updates, or forcing Apple to maintain a private API they want to change or delete but can't because too many popular apps use it.
After all, there are plenty of open source projects on Github that merely use private APIs, the browser for tvOS[0] for example.
I think it's more accurate to say that Apple asked them not to use that method, and they complied. Apple didn't employ any technical measures to prevent the method f.lux was using, you can still sideload apps. f.lux apparently decided it was more in their interest to comply than not to comply.
edit: formating & clarity