Not all users may use Amazon or S3. This means that if we were to build something requiring S3 we'd cut off those users.
Building some kind of generic storage layer with multiple backends (S3/S4/etc) would result in us only maintaining whatever we happen to use for GitLab.com. It could also complicate maintenance too much.