Hacker News Reader

Comment by Legogris | Hacker News Reader

Am I the only one who's pessimistic about this? One of the big upsides of Kubernetes is having portable workloads and provisioning cloud-provider specific resources (whose lifecycles very likely outlive clusters!) in Kubernetes just seems wrong to me. Kubernetes is great for managing, orchestrating and directing traffic for containerized workloads but it really shouldn't be The One Tool For Everything.

Coupling everything together like this just seems to make things less manageable.

IMO infrastructure including managed services are better provisioned through tools like Terraform and Pulumi.

View on HN

The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool.

Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.

This is why Hashicorp needs to accelerate their cloud offering.

Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this.

You sum it up well... In these times you don't want to run a daemon

They used to have a managed Vault offering! But then it disappeared one day never to return.

This issue is to a large extent addressed by Terraform Cloud.

Terraform Cloud runs the executable for you on the basis of VCS triggers e.g. open pull requests, merging to master.

Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.

I think this was the promise of Docker's InfraKit [1] (now DeployKit), to have your infrastructure constantly monitored and adjusted depending on the configuration you pushed into it.

Sadly it didn't go anywhere and it's now archived.

[1] : https://github.com/docker/deploykit

At least with many Kubernetes controllers that have been built in the past (I haven't tried ACK), they don't respond to or attempt to "live-correct" changes in configuration which are made outside of Kubernetes (e.g. via the AWS UI). They act on events that are fired in the cluster when a resource they monitor changes in the cluster.

So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never.

In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc.

In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted.

(Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.

I haven't checked the ACK code to see what they do, but this is what the resync period is for in controllers.

Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files

I'm not 100% sure about the specifics for ACK, but I've been using Weave Flux to manage manifests in my cluster. It resyncs every 5 minutes or so and will correct any drift on the manifests that it can see in the git repo. Rerunning terraform on a set schedule can achieve the same thing with pieces of infrastructure that falls under terraform.

The issue with both of these is that if you create any new resources/manifests outside of git, they'll be invisible to the tools. If ACK can solve this problem, I can see it being quite useful, but if not, the problems that 013a mentioned apply.

Yeah always been my thinking with aws or similar operators.

Lifecycle of resources has to be stacked like a pyramid.. I don’t really want the definition of a bucket or a database(!) to live as a crd in a cluster. I’m much more likely to destructively update a cluster than a bucket. Coupling then doesn’t seem to make sense to me.

Some upsertable stateless resources would be nice tho, like having a crd to create an iam role for a deployment would be neat, so I can deploy a workload and define a role and policy attachments using the same mechanism, works where the crd defines the entire scope of the resources, not just create a bucket that can be filled with stuff.

Begs the question of the threat modeling tho, the aws controller has god privileges in the account? If I get to use my iam crd to provision iam resources for me (super helpful) then aws controller needs high IAM privs. So now kube cluster compromise could be escalated to aws account compromise, not really least privilege. Convenient though...

But then also iam isn’t supported so ️

You raise a good point on the chicken and egg scenario of having the K8S cluster created in AWS creating other AWS resources. Rebuilding everything from scratch will take a lot longer since you can't build concurrently.

The advantages of pulumi and terraform over this also come from HCL and the Pulumi DSL _and_ their respective state files.

This AWS controller only has the state through the API objects. Managing that is a pain because the organisation of those objects is left as an exercise for the user or other tools. Pulumi and Terraform constrain that to well thought-out DSLs.

If there was a sane unifying DSL for managing kubernetes objects it would certainly simplify things. It would not, though, solve the speed of rebuilding the infrastructure.

Regarding lifecycle, most larger shops are going to have an infra team that owns “bedrock infrastructure” terraform config for things like VPCs, IAM, maybe project structure. Then each team will have their own terraform scripts for k8s cluster, DBaaS, queues, etc - the application infrastructure.

This tool is great for the latter case; give me a k8s cluster and I can stand up all of my services in a one-shot command, with a single tool chain.

TBD whether you’ll see infra teams moving their terraform scripts into an infra k8s cluster. I’m inclined to believe it’s less of a good fit there, as your bedrock should not change often, so the marginal value over terraform is less.

So the vision behind operators is not what is managed but who is managing. The idea is to move operations and provisioning from humans to machines.

What should exist, is a set of standards that defines generic cloud resources, and than each cloud provider would implement a translation operator from the standard to its own CRD. Of course, the leading cloud provider has no incentive to do that since this would reduce lock in.

For example, there should be a VirtualBucket CRD which would map to AWS Bucket, Azure Container etc.

Terraform was first into this area, but terraform is still manual in nature.

I think that Pulumi uses that same code generation as Arc.

Reply on news.ycombinator.com