If the app is open source, you're free to compile it with Xcode and install it on your device. That's fine, because the open source nature means you can see everything the app is doing and verify that it isn't malware (and if not you personally, then someone can do it).
But for closed source distribution, barring enterprise distribution, it has to go through the App Store. Allowing anything else opens up Apple's customer base to malware.