Hacker News Reader

Comment by camtarn | Hacker News Reader

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing.

I remember it being pretty irritating to use, though, since it wasn't particularly easy to get Apollo to deploy to a desktop machine in the same way it would in production, and of course you couldn't isolate yourself from the desktop's installed dependencies in the same way. I'm using Docker nowadays and it definitely feels a lot smoother.

This is a nice writeup: https://www.allthingsdistributed.com/2014/11/apollo-amazon-d...

View on HN

I've always thought of Apollo environments as containers before kernel features for containers existed. With enough environment variables and wrapper scripts taking the name of real binaries to populate stuff like LD_LIBRARY_PATH, Apollo makes a private environment that is only _slightly_ contaminated by the host.

Apollo reminded me more of Nix than containers. The wrapper scripts are super Nix-y :)

That's what it was. VM with a barebones deployment system that had a ton of hooks in it.

Really really smart idea that IMO helped Amazon in the 2010s immensely. While everyone else was figuring out k8s and whatnot, Amazon had a good system with CI in place for years.

I wonder how it's fared over time. Amazon was never known for internal tooling in many other places. I hope Apollo is still running strong today.

Anyone remember disco and third-party packages, guam and cmf?

There's been a company wide effort to move everything to Native AWS. While many existing services still run on Apollo, virtually all new services are created on ECS, EC2 or Lambda.

Ooh, I'd forgotten about the wrapper scripts.

And yeah, the other thing that made it work, I guess, was having the machine image be very minimal, very tightly controlled, and very infrequently changed - so you didn't have to worry about things changing all the time due to the upstream distro.

Apollo environments try to solve the same problem, but they're definitely not containers - one cannot depend on any fixed paths. I still bear scars from wrestling with various packages' autotools / glib / Python path dependencies trying and hoping desperately to find poorly / undocumented environment variable overrides to get them to let go of a static, build-time specified path and play nicely with Apollo's environment path shell-game.

I wouldn’t compare Apollo to containers, it is directly comparable to Capistrano + chef/puppet.

Not too much of an update, but they were still using it in 2017.

And as of 2021

Reply on news.ycombinator.com