I imagine that the response times of S3 vs. this Frankenstore model will be much better, which may be an issue for certain applications.

Also you need to host infrastructure software that knows where your data is sitting, how to deal with provider failures, how to efficiently route requests, etc. which means yet-another-thing-to-configure.

Finally, if the volume of data you're storing is so expensive on S3, I have to wonder why you have all this non-revenue generating data stored in the first place. Processing it also seems more expensive now because the free bandwidth you get from EC2<->S3 won't apply in the Frankenstore model.