Investigating Multi-Account IAM Issues in S3 and CloudFront | Hacker News Reader

Investigating Multi-Account IAM Issues in S3 and CloudFront

kevinslin.com

43 pointskevinslin25 comments

Cool write up. Just went through this exact use case a couple months ago. I wish there was a simpler way of just making the bucket owner automatically own anything put into that bucket to simplify use cases like this.

there is the next best thing - making sure that objects uploaded to your bucket also give you full ownership over said objects: https://aws.amazon.com/premiumsupport/knowledge-center/s3-re...

For anyone else that is looking at rolling your own static hosting like this on AWS specifically (including CloudFront cache invalidations and protected domains), I highly recommend checking out Amplify Console https://aws.amazon.com/amplify/console/. It's a little bit more feature-rich and avoids headaches like the article ran into. Closer to Netlify than it is to CloudFront + S3.

Does Amplify Console support the multi-account usage pattern described here? That's the part responsible for most of the complexity + headaches in this setup.

Yep, and it does so pretty easily. You just connect Amplify Console projects in each of the two accounts to a particular branch (or branches if you want feature branches in dev) of your git repository.

multiple account support in amplify involves configuring amplify in each account to a particular branch. amplify doesn't support CI/CD types of deployment - instead, you would need to push to a 'dev' branch to triggers a dev build and then later push to a 'prod' branch to trigger a prod build. one reason we went with a central pipeline is to have a single account to manage all deployments as well as control what code gets to production.

Honestly after using Amplify to develop a new app, I have a hard time recommending it unless your use-case fits their design perfectly and will never grow beyond those bounds.

If you've already made any design decisions or need to do something differently than Amplify expects, it is a minor nightmare. It generates multiple local copies of templates for nested cloudformation stacks and doesn't really support manual editing. The basic Amplify CLI commands entirely abstract away what's actually happening with the AWS APIs. "Ejecting" takes loads of time to basically reverse-engineer the Amplify CLI, clean up the generated templates, and then create your own build process that you can tweak.

It would be much easier to recommend Amplify if it wasn't so far removed from the underlying AWS APIs and it provided an easy way to modify those API calls or templates instead of being stuck within its guard rails.

Amplify CLI and Amplify Console have nothing to do with one another. Confusing naming, I know.

Classic AWS. I have a SA Pro cert and I still couldn't reliably articulate the difference between Lightsail, SAM, and Amplify CLI. It feels like they keep adding to each without any apparent strategy. I didn't even know there was another product also branded "Amplify" separate from the CLI.

AWS really needs more centralized decision-making. It feels like individual product teams are allowed to go way off in the weeds without any frame of reference for whether their solutions make sense when plugged into the full suite.

The Amplify naming is a special kind of AWS classicality. Amplify, what a great Netlify like experience! Oh... not that one, you mean the CLI/Framework. Also pretty cool, we'll see how it plays out with the CDK. No? You mean the literal Amplify library that makes UI work easier. All of which can work together, but don't necessarily have to.

I can't complain too much, it's a bit of job assurance for folks like myself that specialize in only AWS. Exhausting keeping up to speed on everything though.

+1 on this. wanted very much to like amplify but after using it for a few small projects, decided it was not worth the trouble. it abstracted too much and was too opinioated for what i wanted to do.

the one use case i do like it for is hosting simple static sites (eg. jekyll based sites).

that being said, i do like the amplify javascript sdk, especially the auth library and react components: https://aws-amplify.github.io/docs/js/api

best part is that you can use the components without using the amplify service/console

> A great engineer that I knew at AWS once said:

> > When you see hoof prints, think horses not zebras.

Looked this one up and the original medical aphorism dates back to the 1940s and was 'When you hear hoofbeats' [1], in case anyone's curious.

[1] https://en.wikipedia.org/wiki/Zebra_(medicine)

It's worth noting that in a computing context the reliability of this heuristic is questionable relative to what it would be in medicine:

https://youtu.be/fE2KDzZaxvE

Zebras All the Way Down - Bryan Cantrill, Uptime 2017

I just learned this same thing (Object ACLs) last night, and my mind was blown that a bucket can have objects owned and controlled by other accounts, which then prevents a third account from accessing those objects even though it has bucket access.

But when you stop and think about it, it's a similar model as Unix file/directory ACLs...I just hadn't expected it to apply to S3 buckets too.

I still have the tab open for the docs to fix access:

https://docs.aws.amazon.com/AmazonS3/latest/dev/example-walk...

author here - curious about interesting AWS issues you've encountered and lessons learned from them

Had a similar issue with CloudTrail logs. These are delivered to your S3 bucket but the objects are owned by AWS's 'CloudTrail account'. This means you can't drop the logs into your security account and then query them with Athena from another account. Took ages to figure that out.

I'm doing/fixing the same thing with the Cost Usage Reports.

It feels like a common pattern to consolidate your logs into a single account for analysis, I wish they made it more straightforward to setup.

Isn’t that one part of what control tower does now?

yep. control tower creates centralized logging, scp rules and more. what used to take a dev a couple weeks to set up can now be done with a few clicks

We haven't really dug in to analyzing those outside of cost explorer yet but that makes sense.

One workaround I've heard of is S3 replication. I've not tried it yet but apparently the replica has the 'correct' ownership.

Not a lot of issues really. Lots of lessons learned and small features I wish existed.

Here's a classic: I had a lambda fed by an even stream hanging off a Dynamo table. The lambda would delete the record after doing a bit of stuff. I didn't realize on my first pass that this would trigger the lambda again... Wasn't a big deal as I noticed the behavior pretty quickly.

yeah, that's the classic infinite lambda recursion gotcha: https://sourcebox.be/serverless-a-lesson-learned-the-hard-wa...

An issue I've encountered with static website hosting on CloudFront+S3 is that if you want `index.html` to be served at the root path of a URL - e.g. https://www.yoursite.com/folder/ vs https://www.yoursite.com/folder/index.html, you must use the s3 website endpoint rather than the S3 Origin. And in that case, unfortunately, it is not possible to use Origin Access Identity.

good news is that you can remedy that now by using lambda@edge to rewrite requests - there's even an aws blog post about it: https://aws.amazon.com/blogs/compute/implementing-default-di...

From the blog post above:

"It does not work on any subdirectory (such as http://www.example.com/about/). If you were to attempt to request this URL through CloudFront, CloudFront would do a S3 GetObject API call against a key that does not exist."

Reply on news.ycombinator.com