When are containers or serverless a red flag? | Hacker News Reader

When are containers or serverless a red flag?

infoworld.com

35 pointsohjeez60 comments

The quality of this article feels kind of dubious. There are ideas here that have some merit but some of the background smells like it was generated.

That aside, I’m a little put off by the grouping of containers and serverless as if they exist at the same level which could warrant a coherent discussion. For example, given that serverless is simply a way to spin up on-demand, ad-hoc http endpoints, the vendor lock-in argument is going to be stronger than the same argument about an OCI spec container.

Maybe I’m in a bubble, but there isn’t much substance here that isn’t already known: using serverless as a crutch is a terrible idea if you are looking to actually scale and containers, well, exist.

It's a submarine article for Deloitte Consulting's Cloud Practice.

The writer of the article is affilated them and I myself have had Content Marketing teams ghostwrite articles with my name on InfoWorld for "Demand Gen".

You will see tons of "thought leadership" articles this month as June is when infrastructure, networking, and cybersecurity products are released as these releases are timed with industry conferences.

What does the author mean with containers? This articke seems little content. A container is basically never a bad idea. Serverless will drive how your application behaves and how it is written and seriously affect anything else. However, containers is just a matter of deployment and what substrate is used. I can deploy an application into a container on my vps or any of those hosted solutions out there. The article groups these together unnecessarily. I agree with the matter of vendor lockin though.

> A container is basically never a bad idea

Eh.

If I'm running nginx on Debian I have to worry about Debian and Debian's version of nginx. If I'm running an nginx container on Debian I have to worry about Debian, Debian's container manager, the runtime nginx is containerized with, and the version of nginx in the container.

The release cadence of Debian is well-known and can be planned around. The release cadence of an nginx container is not well-known, and is another thing the intern can configure wrong (I still shudder every time I see shops use "FROM foo:latest" in production, which is shockingly common). Not to mention the addition of an entire new class of supply chain vulnerabilities to worry about/mitigate (yes, I can build my own container, but I can also just build my own .deb file).

You can use Debian stable as a base image and run its nginx everywhere where there is Linux. The host OS doesn't matter much if everything is containerized.

Except the host does matter, that's the thing, because you're still ultimately running on it. I remember this really bit me once in the RHEL 6 to 7 shift when the syscall semantics changed. Our brilliant plan of keeping our applications in RHEL 6 containers and running them on 7 while we did the upgrade, but everything realtime failed (these were audio processing engines) because of that ABI change. Not to mention the fact that your stack is only as secure as its least secure element, and now you've got two (or more) platforms to keep an eye on for that.

Wait, did you happen to work at Citrix a few years ago? I ran into the exact same problem, also working on audio processing engines :)

No, a flight simulator manufacturer, but we ended up not being remotely the only team it bit (for that matter it may have been you guys' mailing list trail we followed to find the kernel tweak to work around it since Red Hat's answer was "yeah don't do that")

The downside of containers is that you have whatever kernel is on the host. I avoid running Docker containers on bare metal for that reason. You can still have RH6 VMs running containers hosted on a RH7 server.

All this conversation reminded me of Solaris containers having Linux compatibility layers that pretended you were running on Linux even though you had a Solaris kernel under it.

Wonder how much of that lives now in Illumos and other descendants of OpenSolaris.

The host matters a lot in container/VM escape attacks.

I actively avoid running workloads on bare metal and have been doing so since the early 2000’s. It’s always easier to move a VM image to a new generic VM host than to reinstall everything from scratch.

What Docker does is to make delivering the app easier (it’s less involved than a full VM) and also automate the build process with a simple tool (Docker build).

> It’s always easier to move a VM image to a new generic VM host than to reinstall everything from scratch.

Ansible?

This is my backup solution on machines that don’t hold data.

Rebuild from playbooks, and/or Docker compose.

You don’t move the data with Ansible. You can move a VM to do load shifting.

That's the thing.

All containers are, are a standardized way to ship (distribute) your artifact (application) with it's dependencies inside it. It's a tar archive with a manifest file. I find it easier to think of them as RPM/deb files. You get an artifact repository for free and benefits of the onion file system to reduce size.

People package entire OS distributions in their container, I consider that an anti pattern. Unless you are shipping static HTML your container shouldn't have NGINX or any other stuff in their, especially if you are running NGINX in front of your container anyway.

If your app is written in Go/Rust/C, anything that produces a static executable, the only thing in the container should be the static executable... i.e. 'FROM scratch', 'cp my-app /bin/my-app`, 'ENTRYPOINT /bin/my-app'.

If you use Java, Node, Python, anything that needs a runtime you can use NIX to build a container with only the dependencies needed to run those, nothing else. If you don't like NIX there's other ways to do it although I'm not familiar with them personally.

All the security issues with containers comes from the junk rather than just your application and only the dependencies needed for that app. If a system library is not in your container you don't need to worry about patching it.

Containers I find are always a good idea, it's a standard way to package/distribute.

How you run those containers, that's another matter with Kubernetes becoming the de-facto because it's Kuberenetes and everyone else is using it. You don't need Kubernetes to run a container, especially if it is just one or a handful of containers.

Here's the dependency list for Bookworm's version of Python 3: libc6, libbz2, libssl3, libatomic1, libdb5, libffi8, liblzma5, libncursesw6, libnsl2, libreadline8, libsqlite3, libtinfo6, libtirpc3, libuuid1, binutils (why is this here? ar, I think?), libctf0, libgcc-s1, libgprofng0, libjansson4, libstdc++6, libzstd1, zlib1g. It does at least finally stop pulling in 2to3 with Bookworm, but that's nearly 2 dozen non-trivial libraries which now either are duplicated (in which case what was the point?) or exist in 2 versions (which is bad) on my server. And that's if I can put down the iron sysadmin fist and make devs pull only from a blessed image on our local registry (I can deal with the occasional shouts of "graybeard!" and "fascist!" but it makes meetings more acrimonious than they need to be).

Now, I 100% agree that an otherwise-empty container is all you need for a Go app, but then what is the container actually doing that I can't just do with cgroups in systemd or runit? The whole point of containers was "Devs! Free yourself from the tyranny of a sysadmin limiting your library choices!" but, well, that's exactly the problem they introduce.

Python is a good example.

You have several teams, each working on their own Python app. Each app is in various maturity levels, some are old in maintenance mode, some are new using the latest libraries.

These teams or other teams also have other languages they deploy outside of Python.

I'm familiar with Python but far from an expert. I think outside of Docker you would need to use Virtual env and maybe publish to a pip repo or package an an RPM/Deb. You'd have to roll your own build/deployment scripts. Alternatively you'd have ansible scripts or similar, maybe packer to build a vm or something. Multi tenant Python install's are bit of a pain which is why I think you'd need virtual env.

If you do use pip that will not work for teams who do not use Python so you'd have a second set of special build scripts and another way to distribute the app.

So in that scenario you have multiple different deployment mechanisms, a bunch of custom scripts etc.

Using a container would give you a common deployment mechanism across all languages. In your scenario though you should absolutely be a fascist greybeard and say "You must use our companies blessed base image before we will deploy your app". Then when a security update comes out you update the base image and send an email out saying "we see your app is using base image v0.0.0, we have released a new image with patched security vulnerability xyz, please update your application to use it by YYYY-MM-DD"

Nix wouldn't be to bad an alternative for a container but it has a few rough edges and getting everyone to build the nix way can be a big ask. Nix would solve the dependency + distribution issue with it's binary cache.

> "You must use our companies blessed base image before we will deploy your app". Then when a security update comes out you update the base image and send an email out saying

Either that or you trigger a rebuild of their images because a dependency changed.

This is one of the very few cases where using “:latest” is the right thing to do.

I maintain a docker image on Docker Hub (rbanffy/vm370) and I rebuild it from time to time so that it picks up the latest versions of the underlying OS. I still need to migrate it from Debian to something more minimalistic so that this becomes less of an overhead.

If you're using Nix to build containers then you can just do nix-copy-closure and avoid the container hassles altogether.

Containers give you an extra networking layer, theoretically, but the way it is implemented is extremely brittle and leaky, so most people don't use it anyway.

I assume the same logic applies if you're using some other way of building instead of Nix, too.

If your app is a static binary, why use a container at all?

Admittedly running a binary is quite simple to run on a monolith, depending on size and scaling needs a container still makes sense IMO

We just had a problem with caused by docker, and it was exactly the problem that docker was supposed to solve. I find Docker to be a net positive, but it is far from perfect.

> If I'm running nginx on Debian I have to worry about Debian and Debian's version of nginx.

This is kind of a funny example to me because it’s one of the reasons I liked containers so much when I first encountered them: Debian’s packages are notoriously old, and anytime I wanted to run a newer version of something I’d wind up with some patchwork of configuration management code and shell scripts. In contrast, a Dockerfile provides a simple and clear way to run whichever version I want without messing with the base system, and even multiple versions of a given package with conflicting dependencies on the same machine without any fuss.

And that's why Docker is a great thing to be sitting on a developer's laptop and making the next version of something. The insanity was when people looked at that and decided it should be the normative way to deploy in production.

When you use containers, you need to handle building and pushing them in your CI, setting up a registry, and facing the barriers to debugging/troubleshooting them when deployed or maintaining and operating the orchestration if you insist on having more control. If you do these with infra-as-code it's a lot of work and maintenance. If you set these up more manually you end up with the other kind of manually configured infra problems.

Also you need to worry about provenance of base containers, are they up to date wrt security fixes and are they trusted, etc.

Ultimately these are the same challenges you face using any other method of deployment. You still need to build and push artefacts from CI, orchestrate their execution, and ensure your dependencies are secure. Containers really just standardise that, at the expense of some bloat.

Keeping it simple is what matters. At the base level you can treat a container very much like a single binary if you want to - build it, SCP it to a server, and add a systemd unit to run it. But you have the advantage (and some costs) of mostly isolating yourself from the runtime environment.

The orchestration frameworks are where I start to get itchy about complexity and would definitely put that process off as long as possible.

The things I listed are additional things, they don't replace what you need to do anyway. You could be deploying your Go binary or Clojure JAR on that vm/vps, or you build additional complexity on setting up & terraform authoring your registry and it's setup, making it accessible from GH actions, scanning container base images for vulnerabilities, jumping hoops to debug it because it's in a container, etc.

Using containers without a registry feels like a iffy middle ground. You lose the standard workflows and can't use it on container native platforms.. But maybe it makes sense sometimes.

It has upsides too, sure.

If you use DigitalOcean, setting up a registry is two clicks, and CI is an out of the box feature for many CI providers, e.g. [0]. If it's not, or you're using a provider where you need to do more work, well you need to do that work for any other deployment method too.

[0] https://github.com/marketplace/actions/digitalocean-app-plat...

It’s annoying, but it’s not that hard. If you use Dolly, you simply run a standard container.

> Hey man, we couldn't figure out how to do software dependencies correctly, so we put a Linux distro inside your Linix distro.

> Now you have fifty operating systems to upgrade instead of one. Enjoy.

Using nix-copy-closure basically solved all my problems with one command.

I love Guix, I'm veeeerrry slowly pulling a resisting dev team by their hair into using it, but it also doesn't solve this problem. I don't know if Nix has a lint like Guix does, but when I see the actual number of different versions of a library I have sitting around on disk being run by something (by what? there are ways to find out but none of them are fun) my gray beard gets grayer.

I mean the whole point of nix(and guix probably) is to accept that this is our reality and then built tools that make it tractable.

For me, a technology itself isn't a red flag. Not being able to explain the reason for choosing one approach over another is a big red flag.

Or if the reason is "I want to try this new thing" and not "Our standard production stack won't work for this because (valid technical reasons)" that's a huge red flag.

Some combinations tend to be red flags. Most Windows server setups are, for instance. I find it alarming to see PHP or Django applications deployed on IIS.

There are some situations where Windows may be the best answer - compliance and fleet management, for instance - but unless you are running a Microsoft app that requires (or works better with) Windows, it is a lot of extra attack surface.

I would be immediately suspicious of an IBM mainframe or an IBMi lying directly exposed on the internet as well. I know web servers run under IBMi and zOS, it’s just that it seems like a crazy choice - that’s not what those machines are designed for.

As for that being a red flag rather than a suspicious choice, my attitude is always to find a reason for it - just assuming idiocy is both unkind and arrogant - there may be valid reasons for the craziest choices.

As an aside, in the late 90’s a Brazilian ISP and portal went proudly with Novell’s Netware. They died without a trace after numerous outages and nobody was really surprised.

If you are doing something nobody else is doing, you are either crazy or very clever and the odds are almost never favouring clever.

Not a great article but the underlying point has merit. My personal framework:

- Always use containers, unless you can’t

- Never use serverless, unless you must

And above all, regardless of what you choose, keep the complexity to a minimum. Script and automate everything, eliminate manual processes, and never use two services when one will do. YMMV.

Containers, like static linking, protect you from someone else’s platform if you can’t rely on it. But if it’s your own platform, why can’t you rely on it? Who’s maintaining it?

Best summary of containers - clear and hits the point.

Maybe you just want the ability to run on any platform.

> But if it’s your own platform, why can’t you rely on it? Who’s maintaining it?

I don’t know if this is a rhetorical question. You are now responsible for maintaining your custom environment.

Obvious reasons why you can’t rely on maintaining your own platform include a lack of time for maintenance, a lack of understanding of the platform you have built, and a lack of prioritisation compared to other requirements.

A huge amount of the value that senior admins provide is simply stopping junior and mids from building custom platforms usually because they’re excited about a technology rather than aware of the wider picture.

> a lack of understanding of the platform you have built

There is also the common extension to this problem: someone else built this platform some time ago and has long since left the company.

Yes exactly! Or “this has a bug, is only understood by one person and he’s busy right now”.

Yeah, when I say “our own platform,” I’m thinking something like “we run Debian bullseye 11.7 in prod, build a .deb for that.” Starting our own unique distro in a Docker container is not a good use of time.

Discussing tech choices early on is always going to happen. You can try and be a purist and tell everyone to just talk requirements, but people are people, and requirements and tech are intertwined. An unreasonable requirement may suddenly make complexity balloon, but if early on it's spoken out loud that requirement XYZ is very difficult with technology ABC, it may emerge that the requirement was only a nice to have and can be lived without. So it goes both ways.

There are still mitigations around this, just my opinion. So running in containers means you have a reasonable level of confidence that it can run in many places. How you run it in containers may make a difference. Standalone on its own represents a very small investment of overhead, but it's on a VM, still not terrible. k8s is a popular way to do it but is massive, and is not that different from a lock-in. Fargate and Container Apps take away a lot of that overhead and the lock-in is minimal, because you can escape to other kinds of hosting quite easily. I'd pick Fargate over the other choices where possible.

I tackle this problem by focusing on building the product, not the infrastructure and making sure the first version works just for me.

Once you hitch your wagon to anything you're along for their ride and have to make sure you can still do your own thing while wrangling your dependencies. So choose things that don't move very fast and don't ask much time from you

I can’t talk much about serverless, I briefly tried some frameworks during the hype and I haven’t been convinced. It did sound more annoying and expensive than copying files using ftp to cgi-bin or mod_php managed servers like in the 90s and 2000s.

About software containers I think that the complexity is very very much worth it. Being able to ship software that run on my machine and will run about the same in production is so convenient. Running applications with conflicting dependencies is not a problem. Installing up to date or legacy software without breaking the operating system is also trivial.

Sure kubernetes is complex but it’s a bit more than running containers. A good start is using Docker or podman directly. You can see the value of kubernetes or nomad later.

The concept of “serverless” is just based around trying to recreate a cgi from a long-running process application. Like you pointed out, we have had this since mod_php.

Except we didn’t have spend a day to get the IAM permissions just right and embed some javascript into a huge blob of YAML. Simpler times.

Also, less secure times

Maybe in general, but that does not encompass every project or organization utilizing web technologies. There was technical know-how but often just a lack of willingness or understanding from higher up that prevented a security-focused approach to building applications.

Serverless is CGI for dynamically allocated server clusters managed by other people.

We didn't have that.

From the user’s point of view (someone developing an application), we did via shared hosting and cgi-style scripting. It doesn’t really matter what tech powered the infrastructure as long as it worked. Putting PHP scripts on in a folder to be called at will to perform specific actions was a pattern utilized to great effect with the additional benefit of not having to wait for the function to spin up.

Wasn’t the creation of mod_ the ability to keep the interpreter open?

Yes, before that, we had stateless scripting in web apps. Like serverless it was far easier to debug. Unlike serverless it was slow.

In my experience, you gain enough knowledge to set up a new container service in minutes fairly fast. I opt for containers because to me it's the perfect balance of ease of use, maintainability and performance. Managing container orchestration is completely optional too. All cloud platforms offer it as a service. With serverless I've had more trouble than benefits. It's not easier than containers to set up - often quite the opposite. When (not if) you encounter performance issues, they are either difficult to solve or simply impossible. I have never had this experience with containers.

> Container and serverless platforms introduce additional complexity that the teams building and operating these cloud-based systems must deal with.

This is bizarre. Cloud functions are really the easiest way to deploy code to the internet. Compare that with the other extreme of building a house to host physical servers, hiring staff to maintain them, picking hardware and operating systems, creating procurement and maintenance procedures to keep both up to date. You have to deal with considerable amounts of HVAC design, hiring contractors, etc.

On the other side, people can just write a few lines of Terraform, swipe a company credit card and deploy to millions of users. Not all technology is complicated.

You're conflating complex and hard. Cloud functions might be easy, but they are also complex, and there are definitely specifics you need to keep in mind when developing them.

There's a lot of complexity in the stack that makes a web server run. Nobody would suggest designing their own chips to do so. People insist that cloud computing is adding complexity when it often is a useful abstraction that allows companies to concentrate on the stuff that provides business value vs having to deal with lower levels of the stack.

The sibling post already makes the same point, but I absolutely agree that serverless is specifically less complex than maintaining state for monolithic applications.

Individually deploying endpoints in seconds is hugely beneficial.

Likewise functional purity is easier to reason about.

I find the actual deployment with cloudfunctions easy as you say.

The complexity is the orchestration, the architecture, the hard vendor limitations, having to deal with being a distributed system.

Having a FAAS service makes the deployment easy but development wise a traditional monolith deployed on static servers can be far simpler to deal with.

I use FAAS (Lambda's /cloudflare functions) a lot for pet projects as it costs me fractions of a cent to host. The architecture is quite complex being distributed, dealing with restrictions, eventual concurrency etc etc. I find it enjoyable making things work in these systems basically for free but if these projects where a business with paying customers, alot of the stuff I'd drop from FAAS and deploy on static VPS servers as architecture is so much easier meaning I'm much more productive. With paying customers I can take the additional cost of a few VPS's or dedicated hosts such as Hetzner. The FAAS wouldn't outweigh the productivity gains.

> Cloud functions are really the easiest way to deploy code to the internet.

I have to agree. Anyone who wants to see what the SoTA UX looks like should go try an Azure Function Quick Start guide (preferably v4/C#), use Github and just "let it happen".

I watched my project manager set up the CI/CD pipeline by clicking 1 button, confirming GH org, and setting 3 DDLs. It's all done now. We push code and it goes live. Someone set up a custom domain with 2 extra DNS records and 3 clicks in the portal. No TLS certs to manage. It's all just handled. Anyone who is complaining about this stuff being hard or broken in 2023 is doing the intentional stick-in-bike-wheel meme.

I found the experience to be so simple & robust that I insisted on it being the only experience we have moving forward. You can really build anything directly on top of an HTTP trigger function if you are clever enough and are willing/able to step outside of certain "norms".

From an audit & compliance perspective, I don't see how the word "complexity" would be permitted to enter the lexicon. That is, unless we are talking about the complexity faced by your cloud vendor's compliance team. One massive advantage with serverless platforms is that - ostensibly - there is no sever. This eliminates entire libraries of audit items from your side of the fence.

This article feels like it was written by a disaffected '00s developer who misses Capistrano.

The point of this article is to associate containers (a foundational technology) with "severless", which peaked as a search term back before the 19 in COVID19.

If the company isn’t a cloud provider and they have their own k8s installation.

Reply on news.ycombinator.com