One problem is that S3's object availability is only 99.99%, which sounds good, but if your site has 20 assets it needs to load (html, js, img, fonts, whatever) then 1 out of every 50 non-cached loads could be expected to be missing an asset which can be a big problem for JS heavy sites.

I'm not aware of any invalid status codes, the main issue is that vanilla S3 does not support index or error pages. Instead, it either returns Access Denied or a list of all objects when making a "GET /" request. However, S3 has a website feature, which allows you to set up index and error pages, and redirects.

You might also want to put CloudFront in front of the S3 website. You lose some capabilities like root domain, but you gain a lot of performance and voodoo magic like mapping part of your website to a different origin (e.g., you can request RSS feeds directly from a third-party website using only JavaScript and CloudFront as a proxy).