Hacker News Reader

Comment by rjurney | Hacker News Reader

I can’t really imagine going back to scripting servers remotely to boot up without docker. It sucked. And if you’ve got docker you want something like ECS, cause why manage your own servers? It’s a bit rough to learn but man life sucked before PaaS. You had to build your own. Now I write a simple install script for docker on my machine, edit a menu and as many servers as I want run that app.

Otherwise I dig.

View on HN

Author here. Docker is the only one in that list I’m on the fence on. However, I feel that the EC2 AMI can be the equivalent of the container image, and Docker would only be adding another layer and another OS to deal with. Sure, the AMI is not as portable as a docker image, but all in all I prefer working directly on the EC2 VM just for the sake of reducing layers.

Very interesting point.

But what about maintenance?

Working in a place where we have hundreds of small apps, the issue of maintaining umpteen servers is a great pain. Docker reduces the footprint of what has to be maintained substantially.

Yes I can see that. I was never in a situation of maintaining more than a couple of apps, so my perspective might be shaped through that.

Curious, what does "maintenance" mean for you?

Patching servers can be part of it. Also fixing minor issues when no major development is being paid for.

FWIW, there is tooling in place such that you can just recycle the hosts in your fleet and they'll be brought up with the latest image. so you can become just another scheduled and automated event.

Don't quite follow the latter.

Stop me if I'm wrong, but there is technically nothing blocking AWS to have the same management in EC2. Scaling docker containers is the same allocation problem than scaling VMs, so it could be done with EC2 too.

Maybe AWS spins up an actual EC2 instance for each container, it's an implementation detail.

EC2 is a legacy tool with different abstractions, but if you replace "VM" by "hypervisor" and "container" by "microkernel-app", you could do the same thing with one less layer and the same tech we always used

What about development-wise? One of the big advantages I’ve found with Docker (at least on my teams) is that we’ve mostly eliminated discussions about things breaking on local dev machines due to environment differences. When things break it’s almost always due to code and configuration within our application itself rather than some OS package being a wrong version or something similar. How do you achieve that just using EC2 AMIs without maintaining remote EC2 instances for each developer and keeping them up to date?

That was exactly the reason I was very close to adopting Docker recently. But then I realized it was a very rare problem, in my experience. Maybe it’s because I always worked in small teams of <= 10 people, or maybe the stuff I work on tends to not cause local dev problems frequently. But I definitely see how Docker can help a lot there.

Even on teams of small people (< 10 people) you can quickly run into issues if you don't develop in the same environment you deploy. For example at one company I worked for, developers would use pypi packages but the operations team required us to deploy using the version of the python package available on the Debian our hosts ran. This would create not infrequent discrepancies that certainly could have been dealt with a number of ways, but would have been pretty simple if we could just bundle a single environment and use it locally and in production. Vagrant and VirtualBox VMs are another alternative (that we did use too!) but Docker allows you to get even closer more easily if teams can agree on it.

Being in operations and as a dev, I can tell you that is something operations people are doing wrong. They make their own life harder. What about dependencies that are not in the repo? Will they be building their own packages? What if one of packages has a bug discovered in production, will they create their own package with a bugfix risking breaking system components that might depend on the old version. What when the OS version is EOL to migrate now your app needs to be tested with the new OS, because it is tied to it. I have seen an upgrade from Python 2.6 to 2.7 being unnecessarily painful because of this, even though Python 2.6 code can run on Python 2.7 without changes.

You absolutely want your application to be disconnected from the system you running it on. Languages like Python or Java were designed to run the same not only on different OS but also on different architecture, why would you be ruining that?

With Python I highly recommend if you use redhat or CentOS to use IUS repo which let's you choose exact Python version you want to use. Then create a virtualenv with the exact dependencies you need. If your ops are standing on the way of this they are misinformed, because what they are suggesting ads a lot of unnecessary work for themselves as well with no tangible gain.

Not even in the same universe. Containers using some managed k8s (EKS, or GKE) cluster with helm is ridiculously easier to manage a dynamic sets of applications running than bare amis. I have a k8s cluster that runs across >1k ec2 instances where I have thousand of containers running various jobs. I can apply/delete any massive deployment across that literally in seconds. with a single command.

The layers make it a lot lot easier here.

My perspective is shaped by my experience developing and maintaining mostly monolithic apps on fleets of servers. I never had the problem of having to manage dozens/hundreds of independently deployable jobs/services. And I hope I never will :) I can see how K8 can be super useful there, but it’s a situation I try to avoid :)

You know, that makes sense. I was missing that context.

If you ever do, please give container orchestration a look. It makes it honestly easy.

Weird -- the services you're utilizing are developed exactly the way you're going to avoid!

When developing a monolithic application your choices make sense. When you're taking an approach designed to allow for feature development by multiple teams in an Enterprise setting they start to be a bottleneck.

I’ve seen AWS from inside and I helped build a small part of it. Each individual product is closer to a monolith than a constellation of indispensably deployable jobs/services. The 3 products I worked on were definitely monoliths because we intentionally wanted them to be so. Obviously I don’t mean that they didn’t horizontally scale. Only that the entire app could run from 1 process, the build system produced 1 build artifact, every deployment deploys to all server, and there’s only 1 version of the app running in prod.

K8S and containers in general encourage people to split applications into micro services even if there is no good reason for it.

Don't be discouraged by the docker love in tech right now. I'm not saying it's a fad (it's basically critical for local dev these days), but nothing will replace the joy of simplicity and empowerment of working with the Linux virtual machine directly instead of through endless PaaS or compose or k8s abstractions.

If we assume there's a way to quickly build an image for a vm (like docker build), are there workflows to quickly spin up and throw away those images?

Fargate.

1. That's docker. 2. I should have specified "in local development".

I think Packer (https://www.packer.io/) might what you want there.

From a thread a while ago I know of these that work on Dockerfiles but build fully bootable disk images:

https://godarch.com/

https://github.com/imc-trading/dock2box

https://buildroot.org/

https://github.com/ottomatica/slim

There is packer I highly recommend to use it together with salt module to do all the configuration.

There is also NixOS together with NixOps it has some steep learning curve, but I think it is worth it. You edit configuration.nix which is like Dockerfile, and then you can either use standard AMI and provide the configuration, that will result in setup similar to salt, ansible, chef etc. You can also use NixOps to provision such machine, you can then place /etc/NIXOS_LUSTRATE file which removes state after next reboot, stop it and generate AMI that essentially emulates what packer is doing. You can also use nix build to generate an image file and upload that to AWS.

NixOps is also a great tool to test, you can deploy your configuration to local virtual box, or public cloud provider.

With Fargate you don’t need Kubernetes or any crazy abstractions. I went from not knowing anything about Docker to setting up an autoscaling (didn’t need autoscaling, I needed HA) Node/Express microservice with CloudFormation within two days.

For me, modern Linux VMs keep changing with all the systemd stuff. It’s like every time I try to do something basic like configuring networking I have to lookup whatever they’re doing this week. Fargate and k8s seem better documented nowadays and just runs my app.

That’s the second part of the story for me. From 2000 until last year, all of my development and deployments have been on Windows.

I just started developing with .Net Core, Node, and Python last year. My “Linux deployments” were all Lambda. The Node/Express app used the lambda proxy integration.

I needed to move past the lambda limitations - 6MB request body and cold starts - and I didn’t want to introduce Linux servers into a Windows shop or host Node on a Windows server and I didn’t know Linux well enough myself. I was able to deploy a Linux Docker Container on Fargate and have all of the console output to go to Cloudwatch.

The AMI build process is also incredibly slow compared to docker. When worked into the build process, creating AMIs probably accounts for an extra ~30 minutes for every build, as you have to spin up a separate ec2 instance to create the AMI, snapshot it, copy full AMIs around between regions...

I don’t build AMIs on every build. In fact, nowadays, I just use the latest Amazon Linux AMI and set up the necessary stuff on instance boot using a UserData script. Example: https://github.com/encrypted-dev/proof-of-concept/blob/af60b...

That certainly works and is faster. But doesn't it turn UserData scripts into (part of) a provisioning management system? For instance in that example, you could have a machine die in the middle of the night, the autoscaling group replace it, and end up with a different version of node.js than the one you had before. Or the package server goes down and you can't bring up new instances at all.

I suppose you could bake everything into the AMI except your code, and hardcode the git sha into UserData to ensure your builds are reproducible. It just seems like it might get complex when coordinating lock-step code/dependency changes.

I started playing with Nix and recently also tried NixOS and NixOps and I must say that it actually did correctly what other tools (salt, chef, ansible, docker, vagrant, packer and others) failed to do.

Nix approach is to build everything from ground up without depending on anything outside of it. It caches results so you don't have to rebuild the whole system when building your application. This approach makes fully reproducible builds[1], because the entire environment is known.

Nix by itself can be used for deployments, you can install it on any machine that has nix installed and don't need to worry about any other dependencies, since nix will take care of all of them. It can generate a docker image of you need it, and it will only contain your application with is dependencies. You can use nix-shell to define CDE with all developer tools installed with exact same versions that way developers only need to have nix installed and nix will take care of all other dependencies you need.

NixOS takes what Nix does and takes it one step further and uses a configuration that similarly describes an entire operating system. The single configuration.nix describes what your system supposed to have installed, and configured. You can either deploy that and have nix configure machine on boot, configure machine create /etc/NIXOS_LUSTRATE file which removes all state on next reboot and create AMI out of it (equivalent to what packer does). Or have nix generate an image file and upload that to AWS.

NixOps supposed to be for deployments, but to me it replaces vagrant and docker you can create configuration.nix and deploy it with local vbox, ec2 and other cloud providers. The great thing is that your configuration file will just work fine no matter which provider you use.

There are some rough edges though, for example I needed to update NixOps to use boto3 so it works with assume role and MFA, I hope it will be merged soon.

I believe the issue is that what they are doing is very ambitious and they have limited number of developers to handle all of the work, but from all devops tooling I used they seem to have the right approach. They provide the immutability reproducibility at the right level and doing it the right way (declarative (through use of pure, functional, lazily evaluated language starting with a fully known state) vs an iterative language with a partially known state)

[1] your need to pin your build down to specific version of nixpkgs

I don't agree life sucked before PaaS. I've been responsible for responding to far too many production issues where the PaaS tooling got in the way of rapid response.

PaaS is a problem for me because so many developers don't understand (or own) the tradeoffs of choosing PaaS, so i have to constantly have the "no-PaaS" argument to people who have never dealt with the responsibility of uptime response and can't fathom why I'd want to "do more work when the PaaS does it for you"

I think if you start a new project, PaaS is definitely the way to go. You don't have that many infra people available, you don't have time to deal with building your own deployment details, etc. Once you have time, staff, and real users - move from PaaS to something you can prove is both a better fit and more reliable than (for example) Heroku.

But in the meantime? PaaS services solved a lot of issues for you for less money than it would take you to interview a sysops person. Use them as much as you can, whether it's complete app hosting, abstracted container deployment, or something else.

There's a third option: as you grow in your career as a developer, learn the ops stuff so you can make good choices from square one.

If your site is simple sure PaaS will work, but anything advanced and even simple things are becoming complicated.

Reply on news.ycombinator.com