The AWS Controllers for Kubernetes
aws.amazon.com
aws.amazon.com
Team is concerned about the security ramifications of setting up IAM permissions from ACK: https://github.com/aws/aws-controllers-k8s/issues/22#issueco...
Look, it'll be great when it matures... but this is very much in the developer preview stage. Caveat emptor.
Coupling everything together like this just seems to make things less manageable.
IMO infrastructure including managed services are better provisioned through tools like Terraform and Pulumi.
Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.
Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this.
You sum it up well... In these times you don't want to run a daemon
Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.
Sadly it didn't go anywhere and it's now archived.
So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never.
In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc.
In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted.
(Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.
Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files
The issue with both of these is that if you create any new resources/manifests outside of git, they'll be invisible to the tools. If ACK can solve this problem, I can see it being quite useful, but if not, the problems that 013a mentioned apply.
The advantages of pulumi and terraform over this also come from HCL and the Pulumi DSL _and_ their respective state files.
This AWS controller only has the state through the API objects. Managing that is a pain because the organisation of those objects is left as an exercise for the user or other tools. Pulumi and Terraform constrain that to well thought-out DSLs.
If there was a sane unifying DSL for managing kubernetes objects it would certainly simplify things. It would not, though, solve the speed of rebuilding the infrastructure.
Lifecycle of resources has to be stacked like a pyramid.. I don’t really want the definition of a bucket or a database(!) to live as a crd in a cluster. I’m much more likely to destructively update a cluster than a bucket. Coupling then doesn’t seem to make sense to me.
Some upsertable stateless resources would be nice tho, like having a crd to create an iam role for a deployment would be neat, so I can deploy a workload and define a role and policy attachments using the same mechanism, works where the crd defines the entire scope of the resources, not just create a bucket that can be filled with stuff.
Begs the question of the threat modeling tho, the aws controller has god privileges in the account? If I get to use my iam crd to provision iam resources for me (super helpful) then aws controller needs high IAM privs. So now kube cluster compromise could be escalated to aws account compromise, not really least privilege. Convenient though...
But then also iam isn’t supported so ️
What should exist, is a set of standards that defines generic cloud resources, and than each cloud provider would implement a translation operator from the standard to its own CRD. Of course, the leading cloud provider has no incentive to do that since this would reduce lock in.
For example, there should be a VirtualBucket CRD which would map to AWS Bucket, Azure Container etc.
Terraform was first into this area, but terraform is still manual in nature.
I think that Pulumi uses that same code generation as Arc.
This tool is great for the latter case; give me a k8s cluster and I can stand up all of my services in a one-shot command, with a single tool chain.
TBD whether you’ll see infra teams moving their terraform scripts into an infra k8s cluster. I’m inclined to believe it’s less of a good fit there, as your bedrock should not change often, so the marginal value over terraform is less.
- start an single application
- deploy a set of interconnected apps
- define network topologies
- define traffic flows
- define vertical and horizontal scaling of resources
And now I can define AWS resources.
This creates an interesting scenario where infrastructure can be defined by the k8s API resources and not necessarily have k8s build it. For example podman starting containers off a K8S deployment spec. It's an API first approach and its great for interoperability. The only downside is managing the yaml and keeping it consistent across the interdependencies.
It removes the down side of yaml all over the place. It's missing the package management features of helm. But I have several jars acting as baseline deployments, and provisioning. It works really well, and I have an entire language. So I can map over a list of services, instead of do templating. The other big down side is a java run takes a minute or two to kick off
I was resilient to k8 for a long time. Complexity was secondary to cost, but Digital ocean has a relatively cheap implementation now. This commonality and perseverance of tooling is great.
I want metrics, a simple annotation. I want a secret injected from vault, just add an annotation. It's also cloud agnostic, so this logic can be deployed any where some one provides a k8 offering.
EKS was very powerful. As running service accounts via non managed clusters. Removed the need to pass an access key pair to the application. That service account just ran with a corresponding iam role.
[1] https://github.com/fabric8io/kubernetes-client [2] https://github.com/fkorotkov/k8s-kotlin-dsl
It appears though that Fabric8 is useful for solo java projects without complex dependencies on non-java projects or small java shop. It overlaps with where jenkins-x is going, which has made major strides in the last 24 months.
The original team that worked on Fabric8 lead by James Strachan all moved on from Redhat and many of them are working on Jenkins-x.
* the magic permissions ghost that runs in Azure whose job it is to inexplicably deny you resources to things won’t interfere
* Said Azure service will stay up long enough to be useful
* you finish writing the insane amount of config Azure services seemingly require before the heat death of the universe.
* Azure decides that it likes you and won’t arbitrarily block you from attaching things to your cluster/nodes because it’s the wrong moon cycle/day of the week/device type/etc
* you can somehow navigate Azures kafka-esque documentation got figure out which services you’re actually allowed to do this with.
It is only a slight exaggeration to say that Azure is the most painful and frustrating software/cloud product I’ve used in a long time, probably ever, and I earnestly look forward to having literally any excuse to migrate off it.
Azure’s enterprise sales machine will make sure your wait is long. They sell bulk block deals to CIO, not to devs directly.
> In 2030 it will rock!
Apart from _maybe_ VSCode, what has MS made that makes you go "this rocks"?
Fair play, I'm either ambivalent or not a fan of most of those - except maybe the Xbox, I have a PS4, but the X1 is pretty nice.
> Azure DevOps...one of the poorly documented services
So like, all of them? Hahahaha
> SQL Server on Linux
Next to Azure, this is the next biggest cause of issues at work for me, and I haven't seen any wild advantage of this over Postgres, with the disadvantage that pretty much every linux/macOS and open source tool I use is much happier to plug in to Postgres instead.
I really think that's unfair, untrue even. I work with a lot of services on a regular basis, and while it very frustrating to find something with missing documentation, it seems to be confined to certain services, and mostly then for new'ish features. In those cases, they are quite responsive on GitHub.
> Next to Azure, this is the next biggest cause of issues at work for me, and I haven't seen any wild advantage of this over Postgres, with the disadvantage that pretty much every linux/macOS and open source tool I use is much happier to plug in to Postgres instead.
I also personally prefer Postgres, and by a margin. But I have also worked with SQL Server on Linux, and haven't found any differences over SQL Server on Windows - it just works.
I was being a bit tongue-in-cheek here, but that is good to know they're responsive on GitHub.
> But I have also worked with SQL Server on Linux, and haven't found any differences over SQL Server on Windows - it just works.
Sorry I meant SQL Server in general, not specifically the linux one. You're pretty much relegated to using ODBC and messing around with all the requisite configs there, which is fine once it's working, but it's never as nice and straightforward as other db's ime.
So never dealth with Oracle outside Java?
I can't speak too broadly but mssql/tedious modules for node don't use the ODBC interface and work pretty well regardless of platform with minimal issues.
My biggest issues with MS SQL Server for Linux vs Windows comes down to some of the features that most dbms don't necessarily support in the first place.
> Next to Azure, this is the next biggest cause of issues at work for me, and I haven't seen any wild advantage of this over Postgres, with the disadvantage that pretty much every linux/macOS and open source tool I use is much happier to plug in to Postgres instead.
It's been a pretty great thing for me, if only to be able to get a better local experience with Docker. A lot of our applications are centered around MSSQL Server (for better or worse), I'll say that despite the costs, replication in MS SQL is night and day better than Postgres.
It really depends on how you use SQL Server because it doesn't support all the features of the Windows version and if software uses those features, it gets pretty ugly. That said, if you develop against the Linux version, you're unlikely to have issues deploying to the full version, or Azure SQL.
Of course on other projects we're using PostgreSQL and I tend to prefer it as binary json support is huge imo. Getting other devs to understand that you don't need to break off child records into separate tables that require series of joins, that's another story.
A big reason is that C# has always 'rocked' and now they've got a platform that can be deployed on linux that feels complete. That's super awesome of them. And their documentation for dotnet core is top-notch.
That said, the last windows 10 auto-update managed to cripple the UI and (for real) network speed of the computer it ran on. That definitely didn't make me feel great about MS.
It's a mixed bag. It's almost as if the company is huge and we cannot say much about it as a whole.
* no good working examples in docs
* docs hard to read
* docs are not consistent with reality
* the web portal UX is inconsistent and outright weird (when you navigate through resource group, you can scroll back horizontally to previous context/screen; what a joke)
* there are a gazillion preview api versions that never gets released officially.
* and if you’re lucky to work with azure devops, it’s like building a house of cards with different card types and sizes
I’ve worked with AWS and GCP in the past. Indeed Azure is often chosen by CIO’s rather than people that have to work with the service every day.
Docs being hard to read and inconsistent with reality is a big point. My favourite mismatch is the storage classes one: it turns out there's actually 2 different grades of SSD available, but their examples and docs only mention premium SSD's. I only discovered "normal" SSD because they happen to auto-create a storage class with them in your Kubernetes cluster. The adventure to figure out whether you can attach a premium SSD to an instance is a whole new ball game - trying to find which instances _actually_ allow you to attach them is like looking for a needle in a haystack. Why are they so difficult about it? AWS is like "you want an EBS volume of type io1? There you go, done". Azure: "oh no, you _can't_ have premium ssd. Because reasons".
I found a few instance types when I went looking, but their interface does not make it easy to figure out which ones are premium eligible, but I do remember the price going up not-insignificantly for a premium capable machine, which feels about like double-dipping if you’re also paying extra for the SSD.
Setting them up is a pain: why do I need a different set of permissions to set up a service connection for every single pipeline?
Why does my manager who has what we can only assume is a _marginally different_ set of permissions just see an approve button for the service connection but I get taken to page asking me to setup a service connection? (Which ultimately fails anyway)
Why does it want to create a Kubernetes secret on every deploy?
It does funny things with the output from Kustomise, and we discovered we had to add parameters in to make it stop adding characters to the end of the generated configs.
Did you once, look at a pipeline? Be prepared to get more emails from devops than an overly-eager marketing department.
If you have a Terraform in your Pipeline, Azure will lie to you about the status of you Apply's: it'll tell the pipeline that the resource was created, when it fact it was not successfully created, this will mess up your TF state.
If you deploy to a K8s cluster, and your application crashes during startup for some reason, the pipeline treats this as a failure, IMO that's beyond it's set of responsibilities: it should take care of making sure the deployment configs get to Kubernetes, application failing to start is the concern and responsibility of the developer, not the CI pipeline.
> Those configs feel like they're longer than a Russian literature novel
Assuming you mean pipelines YAML files, I can't agree - my pipelines are always quite terse, and you can also split them into multiple files if wanted. IME they are no longer than pipeline configs from AppVeyor, Travis etc.
> Setting them up is a pain: why do I need a different set of permissions to set up a service connection for every single pipeline?
IIRC, you don't - I'm sure there is an option when creating the connection to grant permission to all pipelines.
> Why does my manager who has what we can only assume is a _marginally different_ set of permissions just see an approve button for the service connection but I get taken to page asking me to setup a service connection? (Which ultimately fails anyway)
Working with service connections is definitely a PITA, but since you need to have the requisite permissions in Azure, I'm not sure there is a good way of handling things.
> Did you once, look at a pipeline? Be prepared to get more emails from devops than an overly-eager marketing department.
I work with pipelines all the time, and don't recall getting a single email from Azure DevOps marketing people.
Oh sorry I worded that poorly - I mean my inbox is absolutely crowded with emails from devops itself. I'm sure I can turn it off, it's just absurdly chatty out of the box.
> Assuming you mean pipelines YAML files, I can't agree - my pipelines are always quite terse, and you can also split them into multiple files if wanted. IME they are no longer than pipeline configs from AppVeyor, Travis etc.
Yeah I was, how long are yours? I've got a couple here that are bordering on 300 lines which is pretty wild considering I only need to build a container and push it to K8s... Best CI/CD I've had was with Drone: equivalent build YAMLS were probably 20-30 lines, tops
> Working with service connections is definitely a PITA, but since you need to have the requisite permissions in Azure, I'm not sure there is a good way of handling things.
Compared to AWS' permission model, my experience is that it's distributed amongst throughout a number of locations, and needlessly complicated to ensure that team members have the permissions they need.
As to the pipeline length, most of my interactions are npm/node scripts via package.json within various projects . Often even other language projects I'll use node/npm as a task runner which makes for decent glue code. That tends to keep my pipeline files much smaller as it's usually...
1: add tools
2: configure environment injections
3. run npm script(s)
It seems Azure and AWS permissions seem to be at opposite ends of the spectrum. I will say I find dealing with Azure's Storage services (Blob, Table, Queue) to be really nice for a lot of common use cases. At least the tables and queues.
As a random example, I have a pipeline that builds a multi-arch container image for both x64 and ARM and pushes it to ACR, and also builds, tests and deploys a Function App, and publishes artifacts. The main pipeline file is about 75 lines, and it makes use of 3 other files that setup docker buildx, login to ACR, set the build version, that kind of thing. In practice, I find these kind of "setup" tasks often use as many lines as the main file, but it's often reusable stuff, and splitting them out into templates is really helpful, I think.
In total, it's around 120 lines for all the pipeline files. I am keen on makefiles though, so a lot of the actual build script is in them, ensuring we can also build locally too.
Another example is a web app, where it builds, runs unit and integration tests in a container, and publishes build artifacts for a release pipeline to take over deployment. It also has a 2nd job where it builds some Windows-specific stuff in a Windows agent. Whole thing is less than 100 lines.
The simplest pipelines will be less than 30 lines - for example, the web app I mentioned could easily be done in 30 lines if tests didn't run in a container.
I don't mind taking a look at your YAML file if you think another pair of eyes would help (my email is in my profile, and you can anonymise bits of your file if you want).
Yeah, they are embracing more income. Not really that surprising.
I don't think so. We switched most our workloads from k8s to ECS (mix of EC2/Fargate launch types) because clusters were hard to maintain. Now everything is visible in Console and we now have tech support down to the ECS Task level. Before doing that, management pestered AWS about future of ECS. AWS insisted that it is actively developed, and lots of their internal stuff runs on ECS, not k8s.
You need to understand why AWS churns out silly services every month. Tomorrow there will be new HypedTech, and days later new AWS ElasticHypedTech service. It's just good business.
The same issue is with Service Catalog and the open service broker model. AWS ACK is way better in its expressiveness, in comparison. (one crd per resource) It is more difficult to generate them though, compared to simply adding existing brokers.
Do you end up with a "meta-kubernetes" to deploy kubernetes clusters and migrate services between them?
> Do you end up with a "meta-kubernetes" to deploy kubernetes clusters and migrate services between them?
Congratulations, you just discovered the ClusterAPI
Also, I have an allergy to complexity, so I need fewer moving parts for health reasons.
In Crossplane, our primary scenario is to have a dedicated small cluster for all the orchestration of infrastructure and many app clusters that will consume resources form that central cluster. So, clusters with workloads come and go but the one with your infra stays there and serves as control plane. See the latest design about this workflow: https://github.com/crossplane/crossplane/blob/master/design/... I'd be happy to hear your feedback on the design.
Disclaimer: I'm a maintainer at Crossplane.
No, I don't want Terraforms, puppets, yaml files, load balancers, nodes, pods, k8s, chaos monkeys, Pulumies, pumas, unicorns, trees, portobilities, or shards.
I love cloudrun and fargate. Cloudrun has like 5 settings, I wish it had like 2.
Granted even with one container and a single DB I found terraform useful as I don't have to fiddle with whatever DB, container, etc. settings there are every time. I also prefer to not have to memorize or figure out a UI I use once every other month. Too many UIs across too many things.
Disclaimer: I work for Render
While serverless can address a lot of this stuff (the load balancer, DNS, cert management, etc configuration could be much easier or builtin to Fargate services), some of it you can’t just wave away (IAM policies, third party servic configuration, database configuration and management, etc). You need some of this complexity and you need something to help you manage it, namely infra-as-code.
https://issoseva.org hasn’t ever gone down.
I think it's both unfair and ill advised to want that type of simplicity with Kubernetes.
That being said, I don't really understand why this is necessary when there are already a lot of tools out there to manage clusters (other than AWS wanting everything to live in their console)
You just add a db ip address to your app config? like any other solution. States work great.
I posited that this was the benefit in knowing Kubernetes all along, and possibly the ace up GCP's sleeve -- soon no cloud provider will have to offer their own interface, they'll all just offer the one invented by Kubernetes.
>Smart companies try to commoditize their products’ complements.
[0] Strategy Letter V https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ (edited)
"ACK is available as a developer preview, supporting the following AWS services:
Amazon API Gateway V2 Amazon DynamoDB Amazon ECR Amazon S3 Amazon SNS Amazon SQS"
And:
"Specifically, in the coming months we plan to focus on:
Amazon Relational Database Service (RDS), track via the RDS label. Amazon ElastiCache offers fully managed Redis and Memcached, track via the Elasticache label."
In 5 years maybe this will be comparable to Cloudformation, but AWS will likely launch services faster than this adds them.
But what I'm trying to get at is that the kubernetes interface basically is the terraform/pulumi interface, just with a different configuration language (and less built-in templating macros, which is a GOOD thing, see CloudFormation), but better, because it's continuous. You write a configuration (no matter how dynamic you make things, there's one version that's sent out to AWS/GCP/whatever) with terraform/pulumi, and terraform/pulumi make it true once. You write a relatively similar configuration with k8s and it is made to be true once, but also continuously.
As far as popularity goes, kubernetes is insanely popular -- kubernetes was adopted by just about every cloud provider with a speed I've never seen before. Terraform/Pulumi basically took on the work of adapting to cloud provider APIs, not the other way around.
[0]: https://docs.ansible.com/ansible/latest/modules/list_of_clou...
This is true if and only if there is a controller written to achieve reconciliation. Defining a CRD doesn't do anything, it's just a reservation of a path and, maybe, a schema attached.
A controller can be written that listens but does nothing. It can be written to do something, but not listen. The API Server does not care. There is no magic logic engine or pathfinder or query planner. Someone has to write a controller.
All that bare Kubernetes gives you for CRDs is the equivalent of MongoDB.
Regardless of corner cases, this is a step change difference between tools like terraform/pulumi that reach out and call APIs and finish executing compared to k8s which supports reconciliation. A tool with reconciliation is strictly more capable than one without (for better or for worse).
I am very unlikely to pick terraform for any personal projects ever again, imagine being able to literally drop down to AWS SDK or CDK in the middle of a script and then go back to Pulumi land? AFAIK this is basically not possible with terraform (and terraform charges for API access via terraform cloud? or it's like a premium feature?)
At the risk of audience running out of pop corn, Cloudformation if the only thing you really need for Infra as a Code in AWS. Nothing else. ECS Cluster, ECS Service, TaskDefinition, ALB - it's all that's needed to orchestrate containers.
1. OSBAPI is not widely known outside of the Cloud Foundry community it came from. In turn that's because Cloud Foundry is not widely known either. Its backers never bothered to market Cloud Foundry or OSBAPI to a wider audience.
2. It imposes a relatively high barrier to entry for implementers. You need to fill in a lot of capabilities before your service can appear in a conformant marketplace. With CRDs you can have a prototype by lunchtime. It might be crappy and you will reinvent a whole bunch of wheels, but the first attempt is easy.
3. Fashion trends. The first appearance of OSBAPI in Kubernetes-land used API aggregation, which was supplanted by CRDs. Later implementations switched to CRDs but by then the ship was already sailing.
4. RDD. You get more points for writing your own freeform controller than for implementing a standard that isn't the latest, coolest, hippest thing.
It's very frustrating as an observer without any way to influence events. OSBAPI was an important attempt to save a great deal of pain. It provided a uniform model, so that improvements could be shared amongst all implementations, so that tools could work against standard interfaces in predictable ways, so that end-users had one and only one set of concepts, terms and tools to learn. It also made a crisp division between marketplaces, provisioning and binding.
What we have instead is a mess. Everyone writing their own things, their own way. No standards, no uniformity, different terms, different assumptions, different levels of functionality. No meaningful marketplace concept. Provisioning conflated with binding and vice versa.
It is a medium-sized disaster, diffuse but very real. And thanks to the marketing genius of enterprise vendors who never saw a short-term buck in broad spectrum developer awareness, it is basically an invisible disaster. What we're heading towards now is seen as normal and ordinary. And it drives me bonkers.
Are other companies doing similar things? e.g. I would love to be able to set up Cloudflare Access for services in k8s
OAM is an optional feature of crossplane - you don't have to use it if you don't want to
This aside, I think crossplane work is interesting.
[k8s-infra]: https://github.com/Azure/k8s-infra
Excited about this, though you do wonder whether it'll suffer the same fate as Cloudformation: the Cloudformation team finds out about new feature launches the same time that the general public does. If the Kubernetes operator lags behind, you're going to have to fall back to something else if you need cutting edge features.
You can orchestrate entire application stacks (pods, persistent storage, cloud resources as CRDs) using this approach.
What I really want is RDS/Aurora though.
The provisioning time of a deployment and an RDS instance is very different though. This is probably most useful when you’re starting a service up for the first time. This is also when it’s not going to work as expected due to that latency of RDS starting up while your app crashed repeatedly waiting for that connection string.
This would be really nice for buckets and near instant provisioned resources, but also kinda scary that someone could nuke a whole data store because they got a trigger finger with a weird deployment and deleted and reapplied it.
My feelings, they are mixed. :D
I guess the 'kubernetes' way would be to create a generalized object for 'object store', that would be implemented by means of s3 on aws and on azure it would be done as blob storage.
Now with this approach you can only do the common features between all platforms, you would have a problem with features exclusive to aws for instance, or you would need some mapping from a generalized CRD object to specific implementation of each platform.
Say I have an operator watching all namespaces on the cluster. Since operator CRDs are global in scope it makes sense for some operators to be installed as singletons. A CR for this operator gets created in some namespace, and it wants to talk to s3 -- it has to bring along its own s3 credentials and only that CR is allowed to use the s3 bucket? You can imagine a scenario where multiple CRs across namespaces want access to the same s3 bucket.
https://cloud.google.com/config-connector/docs/overview
disclaimer: I work at GCP.
Github: https://aws.github.io/aws-controllers-k8s/
On a different note, recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner.
This book covers most of the topics that you need to learn to get started:
If someone is interested, here is the link :)