Ask HN: Is it a mistake to launch an app that only supports iOS 5.x?
http://stackoverflow.com/questions/9094584/launching-an-app-for-ios-5-and-beyond-only-a-mistake
It's a universal app, so this covers iPhone and iPad. Over 80% of users are on 5.0 or above.
My take on this is that if there is any iOS 5 feature that would make your life easier and save you time, or make your user's experience better, then go for it and don't worry about backwards compatibility. But if there's nothing specific drawing you to iOS5 go ahead and support old versions, it's not that much extra work to test and support older versions.
I say go for it, you'll have a large enough install space as it is with iOS 5, I wouldn't worry about your lost users.