Container technologies at Coinbase: Why Kubernetes is not part of our stack
blog.coinbase.com
blog.coinbase.com
This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase.
It was a nightmare. If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. Kubernetes is really not the magic bullet it's billed to be.
> Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it makes them more difficult at this time)
Oh man this hits home. EKS is an absolute shitshow and their upgrade schedule is (a) not reliable, and (b) incredibly opaque. Every time we did a k8s version bump, we'd stay up the entire night to make sure nothing broke. We've since migrated to cloud functions (on GCP; but AWS lambdas could also work) and it's just been a breeze.
I also want to add that "auto-scaling" is one of the main reasons people are attracted to Kubernetes.. but in a real life scenario running like 2000 pods with an internal DNS, and a few redis clusters, and Elastic Search, and yadda yadda... it's a complete pain in the butt to actually set up auto-scaling. Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful).
Did you run into cold start delays with GCP cloud functions? AFAIK it is one of the reasons many prefer AWS Lambda over GCP
I've experienced this as well. At the last large company I worked at we had a Heroku-like system to run our apps on. That was deprecated for a Docker-based solution. And then _that_ has now been deprecated for a Kubernetes offering. We just ran some Python web apps–we didn't want to have to learn and support an entire system. And here's the thing, most big tech companies I've worked with are all made up of these small, "internal" services, that just want a simple place to run their services.
Helm. I’m staring at 20 pages of Yaml. Most of it seems arbitrary. Crap ton of interconnections. Plus defining all iam roles.
Doesn’t help that devops team keeps writing custom scripts to interact with it.
You can use the external services the same way if they are already setup with things like DATABASE_URL that holds a postgres connection string to amazon RDS, etc. Running k8s doesn't mean you have to move your entire stack into k8s, you can still use hosted services just fine.
In all my time with k8s, I never used Helm. If you don't need it, don't use it and keep it simple. k8s can do a ton of stuff but in reality, you barely need more than the basic.
What issues did you encounter ? I had to spend a week working through it to get familiar with everything but I wasn't experienced with docker previously (I've played with it a few times but never had to setup stuff like custom registry, versioning, etc.)
I did end up in situations where the cluster was FUBAR but I eventually figured out that everyone just recommends rebuilding the cluster over diagnosing random stuff that went wrong during development.
This reads like an oxymoron. 'DevOPS' was supposed to break down walls between teams. You probably have a traditional "Ops" organization, regardless of team naming conventions.
> Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful).
How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
What is the alternative to k8s that does not need people to have any technical knowledge?
To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic.
I am a 1-person venture doing everything from JavaScript/React to maintaining backend infra, and I couldn't have done it without k8s.
I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot.
I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.
And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
If it doesn’t fail just right in kubernetes there might be no logs at all.
I’m thinking on particular about trying to mount filesystems into pods.
I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applications critical to our billing operations, and developers on the team deploy new applications with little or no support. Any of the issues I've hit are an artifact of migrating legacy applications not designed to run in more-or-less stateless environments, which is why the PCGW Community site still runs on its own server (Invision Community sucks).
I really don't see all the issues people have that aren't due to a mismatch of application design vs target environment (and no, it's not monolith vs microservice - monoliths run just fine on k8s; but you should be designing your application with a 12-factor environment in mind) or a misguided notion that you will be drowning in YAML hell (it's real, but you can manage it - and it's directly related to the complexity of the services you are deploying).
PCGW is a great example - installing a new Mediawiki extension, changing a config file, upgrading to a newer MW release is just updating a file or a git submodule and committing. I don’t get paid thousands a month to manage the site for the owner, so I make my time spent as efficient as can be done.
Hardly. If anything at all, it tells about the _team_ and/or the culture of the organisation. In any DevOps/SRE/Opsec culture worth the salt, an immediate blameless postmortem analysis would be performed to help with premortem analysis in future.
DevOps is not about exposing unsecured endpoints. You've got it all wrong son.
Fair enough. I am letting you know though the part where you got mixed up - that is not because _What DevOps has become in practice_. That is precisely because of failings and shortcomings in team culture and/or the organisation that practice DevOps.
I have done Softwaredevelopment for ~15 years, swiched now to infrastructure and build a gke cluster. It was awesome, very logical, nice and easy to use.
Now i read stories from coinbase and don't get it.
This is true of software development in general, if not life itself!
But you're right that declarative systems amplify this issue.
But, if you're approaching it for the first time with no assistance, there are lots of things that can trip you up, and lots to learn. That's not a reflection on k8s, it's just the nature of the large set of problems it's solving.
K8s is succeeding because it's very well designed, has a large and diverse ecosystem, and solves set of important problems that very few other tools even try to tackle. Apache Mesos perhaps comes closest, but it's not quite as pragmatic, and its adoption level reflects that.
Also, because of k8s' scope, many people may not fully appreciate the range of problems it's solving, seeing it through the lens of their own background and focus.
To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often.
What I really wonder is where did you learn k8s, which parts did you learn the most? It seems huge and I would love to be in your position
For people who already know Linux, kubernetes comes naturally because it is pretty obvious.
But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.
For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere.
I mean, you terminate one computer, k8s will figure out that the program should be running and it will be started in another computer. (Including moving detaching/attaching the block device from/into computers, this kind of things)
There is no much secret.
For me it really feels like an (well designed/integrated) API on top of standard Linux technology (ipvs, netfilter, mount, process mgmt, virtual ip, etc) via yaml running your stuff in 1..N computers but you manage it as one big computer :)
EDIT: added more context within "()"
To me as a developer heroku is the gold standard.
Docket-compose makes sense.
Kubernes and helm is a giant soul crushing wtf On getting anything done.
Now I’ll get it figure it out. Angular + exhaust+ typescript was a similar experience after I took a break from front end for a few years. A few months of pain, then it starts clicking.
but this just seems insane for a basic web app. So many different tools needed to get it going. Tutorials all have many steps that don’t apply, or use other odd pieces swapped in.
Oooold presentations (think 2016 and older) tended to talk more about basic building blocks and how they interacted, especially the design involving resources and their controllers working in a loop of "check requested state, check real state, do changes to implement requested state", and how those loops went up from single pods, through ReplicationControllers (now ReplicaSets), then Deployments, etc.
You had to become a master of figuring out the direction of where community was going and stay on bleeding edge.
Some people found this fun. I have many times enjoyed this. When I want to get a boring feature built and hand off to the jr develops so I can spend some time with my kids. It’s frustrating
The old presentations? Ones that remember K8s 1.0? There's one major change (move from Replication Controller to Deployment+ReplicaSet) that doesn't invalidate ~90% of the material, because the core stuff is about how controllers work!
Yet it seems more and more common to me that people don't learn the core mechanism of kubernetes unless per chance they got there writing CRDs :|
Ansible can't give you anything like that. Even if you use Ansible to automate something like network setup, the commands and modules you use will be different between e.g. cloud providers. Kubernetes give you a consistent abstraction layer that's almost unchanged across providers, with the exception of annotations that are needed in some cases for integration with certain provider services.
No. For starters, Ansible playbooks don't allow you to dynamically deploy and redeploy a set of containers in the node that at a given moment happens to have more available computational resources such as bandwidth, nor do teu respawn a container that just happened to crash.
Arguably Ansible only implementos a single feature of Kubernetes: deploy an application. Even so, while with Ansible you need to know and check all details about the underlying platform, with Kubernetes in essence you only need to say "please run x instantes of these containers".
An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.
Kubernetes is designed for a cluster of nodes not a single one.
Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.
Bonus points if you just mount the compose file on an NFS share.
I've wasted enough man months on kubernetes that unless you tell me to manage 1000 nodes this approach will never cost me more time than the time I spent using and learning kubernetes.
I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS without k8s - read their docs, figure out the API.
Maybe i'm an "old man yelling at new tech" but meh i just see very little value in k8s because you inevitably need to understand the layer beneath - linux (k8s is far from a non-leaky abstraction imo), PLUS all the complexity of k8s itself. I do see the value when managing a big and complex infra with 100s of servers or something, but very few people have that problem.
Not sure what you mean here.
Kubernetes is designed to be cloud and vendor agnostic.
And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.
But it’s not. Connecting to EKS is completely different from connecting to GCP. Setting up worker nodes is completely different too, oh and load balancers.
It’s only the last mile that’s similar.
Sure, it's far from drop & replace, but "completely different" is a huge misrepresentation in my experience. Running multi-cloud seemed quite doable.
How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environment if you want to add a new server to your cluster? If your app has microservices how do services discover each other? How do you mount volumes from cloud storage? How do you update configuration? How do you automatically restart failed applications? How do you monitor if your applications are working? How do you make sure the right number of MongoDB replicas are running at all times? How do you view your log files remotely? How do you port-forward from localhost to your Linux server to test your app locally?
A handful of dedicated machines is enough. For example: stackoverflow.
None of these are difficult to answer. You can set up automated deployment/rollback however you want. You don’t care about the dependencies your code is compiled and linux has binary compat. You don’t need to split your app into 1000 unmanageable interdependent microservices. You have enough disk space on your dedicated machine or use NFS. You set up your systemd service file to auto restart your service. Etc etc.
When you have k8 as a hammer everything looks like a nail.
So yes, you'd need a handful of different systems, but "k8s" is hardly a single system once you have to actually manage it, and most of the parts are simpler.
Installing application and routing it with load balancing, TLS etc and support for draining is really simplified, something that used to require annoying amount of time (it got better as Lua scripting was extended in HAproxy, but by that time I also had k8s).
Resource allocation, including persistent storage (including NFS) is so much easier it's not funny, it makes my old years painful to think about. Syslog is not novel but getting all applications to log the same way was always a PITA, and at least with containers it's a bit easier (I still have to sometimes ship physical logs from containers...).
As for monitoring, that's one of the newer and more advanced topics, but it's possible to integrate more complex server monitoring with k8s - it already provides a lot of OS/application side monitoring that really makes it easier to setup observability - but now there's quite simple way to integrate custom node states into kubernetes allowing a quick glance way to check why a node is not accepting jobs, integrated with the system so you can actually trigger from a health-check that the node is in trouble and should not take jobs.
> How do you run an application on a cluster of plain old linux machines?
Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).
> How do you do load balancing?
An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes).
> How do you update your app without downtime?
Blue-green deployment, or phased rollout.
> How do you ensure all your servers are running the same version of dependencies?
Build them from a common image.
> How do you update those dependencies?
Update the Packer template that builds that image.
> How do you replicate your environment if you want to add a new server to your cluster?
Start the server from the same image.
> If your app has microservices how do services discover each other?
Consul, or DNS, depending on your appetite.
> How do you mount volumes from cloud storage?
It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.
> How do you update configuration?
Either update Consul and have it propagate configuration, or update a configuration package and push it out.
> How do you automatically restart failed applications?
Systemd restart policy.
> How do you monitor if your applications are working?
From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.
> How do you make sure the right number of MongoDB replicas are running at all times?
Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group.
> How do you view your log files remotely?
Cloudwatch, Syslog, SSH (depending on requirements).
> How do you port-forward from localhost to your Linux server to test your app locally?
SSH.
Also, haproxy is one of the most reliable software I've ever used.
I wouldn't call it the best way; Rather a good way because Kubernetes does encapsulate the really good bits from scalability, development, security and reliability aspect. It's not a panacea but if you have team bandwidth to run k8s cluster, it's definitely worth a look.
<3 HAProxy. It's a solid piece of software tested to the teeth over the years and is great; Here's the thing: You can run it as your preferred ingress controller too :) https://www.haproxy.com/blog/dissecting-the-haproxy-kubernet...
AMEN to this one.
Very nice answer. It's the way we run infra.
Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wants to "do ops", and less so for the dev that already knows ops (or any regular sysadmin/ops person), because they already know all these answers.
"Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!"
No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.
Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day.
CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exclusive).
Yes. I would agree to your statement precisely as an answer to @jen20.
Some things such as getting stateful systems, HPAs and persistent storages were a little tricky initially but a breeze after.
But I do want to mention that you really really need a team to look after it. Without it, it'll bound to be another snowflake.
[edit]: <sigh/> i meant to say stateful when i wrote stateless.
>> How do you run an application on a cluster of plain old linux machines?
> Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).
How come it is any different than running the Docker image on Kubernetes? The image build process is the same, delegating running the image to the platform is the same, the only difference at this point is the name of the platform you are running. Even if you were deploying ZIP archives to Elastic Beanstalk, if it doesn't work as expected, you'd have to debug it as an outsider, and you'd still have to know about the technology. I don't see how it is any different than Kubernetes.
>> How do you update your app without downtime?
> Blue-green deployment, or phased rollout.
How exactly? There are gazillion ways of doing them, they are rough concepts, what we need is a reliably working setup that requires as much effort from us as possible, and there are absolutely no standards on how to do them. Are you going to use Ansible? Maybe just SSH into the node and change the symlink? Maybe some other ways?
>> How do you replicate your environment if you want to add a new server to your cluster?
> Start the server from the same image.
How do you do that? You'd either do that manually on AWS console, or build some tooling to achieve that. If you were to do that via the autoscaling options the vendor is providing, then it is no different than Kubernetes: if that doesn't work then you'd have to debug regardless of the platform that is managing the autoscaling.
>> If your app has microservices how do services discover each other?
> Consul, or DNS, depending on your appetite
What is the difference between trying to learn how does Consul handle service discovery vs how does Kubernetes handle it?
>> How do you mount volumes from cloud storage?
> It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.
Would you mind sharing examples that are not vendor-specific and that'd be configurable on a per-service fashion easily, hopefully without writing any code?
>> How do you update configuration?
> Either update Consul and have it propagate configuration, or update a configuration package and push it out.
How is this any better than pushing your changes to Kubernetes? I personally don't know how does Consul work or how to update a configuration package and push it out to somewhere, I don't even know where to push them. In this context, learning them is also not any better than learning how to do them on Kubernetes.
>> How do you automatically restart failed applications?
> Systemd restart policy.
So, this means that you'd need to learn how to utilize Systemd properly in order to be able to start running your application and write the configuration for that somewhere, and also deal with propagating that configuration to all the machines.
>> How do you monitor if your applications are working?
> From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.
The question was not really that. The tools like pingdom won't help you if an internal dependency of your application starts failing suddenly. You need a standardized solution for gathering various standard metrics from various services of yours, things like request rate, error rate, request durations, as well as defining custom metrics on the application level such as open database connections, latencies on dependencies, and so on. You will definitely need a proper metrics solution for running any serious workload, and in addition to that you'll also want to be able to alert on some of these metrics. There is no standardized solution for these problems, which means you'll need to roll your own.
>> How do you view your log files remotely?
> Cloudwatch, Syslog, SSH (depending on requirements).
The proper alternative to the Kubernetes' solution is Cloudwatch, and even then the simplicity of `kubectl logs <pod_name>` is still better than trying to understand how Cloudwatch works.
>> How do you port-forward from localhost to your Linux server to test your app locally?
> SSH.
This is not a trivial setup. Let's say you have a service A running remotely but it is not exposed, meaning that you cannot reach it from your local machine, and you'd like to be able to use that while developing your service B locally, how would you set this up in an easy way?
The points regarding the images are the same points as any Docker image, so it really boils down to the choice and one doesn't have an advantage over the other in this context.
What I am trying to say is: there are quite a lot of problems when running any kind of serious workload, and there are thousands of alternative combinations for solving them, and they were solved even before Kubernetes existed; however, there were no standardized way of doing things, and that's what Kubernetes is allowing people to do. There are definitely downsides of Kubernetes, but trying to point specific examples like these don't help as they are just names of individual software that also have learning curve and they all operate differently. I do wish there was a simpler solution, I wish Docker Swarm succeeded as a strong alternative for Kubernetes for simpler cases as it is brilliant working locally, and I wish we didn't have to deal with all these problems, but it is what it is.
As of today, I can write a Golang web application, prepare a 10-lines Dockerfile, write ~50 lines of YAML and I am good to go: I can deploy this application on any Kubernetes cluster on any cloud provider and have all these stuff defined above automatically. Do I need to add a Python application alongside: I just write another 20-lines Dockerfile for that application, again ~50 lines of YAML for Kubernetes deployment and bam, that's it. For both of these services I have automated recovery, load balancing, auto-scaling, rolling deployments, stateless deployments, aggregated logging, without writing any code for any tooling.
If you were to deploy a solution like you described, you would get something more complex than simply running Kubernetes, except worse. I suspect that you believe your solution would be simpler only because you are more comfortable with those technologies than with k8s. The more I read criticism of k8s, the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with".
On top of that, you'd need to separately document everything you do on your infra. The advantage of Docker images over AMI is that you have a file that describes how the image was built. With an AMI, you would need to hope that the guy who created the AMI documented it somewhere (or hope that he has not quit). Same goes for k8s, where configurations are committed into your repository.
At the end of the day, k8s stays a tool that you should use only when needed (and also if you have the capabilities of using it), but I think you shouldn't discard it simply because you are capable of producing the same result by other means. You get a lot more by using k8s, in my opinion.
https://packer.io - how is this even a discussion?
> the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with".
My infrastructure runs in Nomad on bare metal. I am by no means opposed to “progress”, I just don’t think Kubernetes is the be-all-and-end-all of infrastructure and would like to have a less hysterical debate about it than the parent to my original post presented.
To be fair though, that's not "Plain old linux" like zaptheimpaler suggested was somehow possible. That's linux plus AWS managed services plus software from Hashicorp. Which is a great stack to be on, but has its own complexities and tradeoffs.
Using plain old Linux and Erlang/Elixir as a backend you can get extremely far with scaling with very little overhead and issues.
It is unfortunate that so many ignore the benefits of using Erlang as a backend technology.
Thankfully, `jen20` was kind enough to take your questions in good faith and made a good demonstration of this confusion.
Also, see jen20's response.
If your only tool is a hammer (called k8s) then I'm sure everything really does look like a nail...
> How do you do load balancing?
With dedicated load balancers. They are even cheaper than servers because they are optimized for doing one thing.
> How do you scale up and down?
With pretty trivial automation than trigger deployments when load is too high.
> How do you update your app without downtime?
(app?!) A package manager pulls updated packages, deploys them and restarts the services. The LB does the flipping.
> How do you roll back easily if something goes wrong?
The package manager does that quite easily
> How do you ensure all your servers are running the same version of dependencies? > How do you update those dependencies?
Package managers support versioning on dependencies.
> How do you replicate your environment if you want to add a new server to your cluster?
You install MyStuff version 1.2.3 and everything goes in.
> If your app has microservices how do services discover each other?
Thankfully Amazon uses services that are not micro. DNS does the trick.
> How do you update configuration?
It's in the packages.
> How do you automatically restart failed applications?
OSes do that since decades.
> How do you monitor if your applications are working?
LBs check the endpoints, monitoring tools do the rest.
Essentially you can build all of this with the technology that existed 15 years ago. It works reliably and it takes much less learning.
I guess if you pay for a hosted service a lot of the control plane is taken care of.
I’ve used k8s in orgs where it’s a great fit and really fills a need, but it is considerably more complex than running a web service balanced across a couple of machines, and it definitely requires a lot more upfront complexity (as in, you have to solve problems before you have they are actually problems).
Paying for a managed provider. Heroku, Elastic Beanstalk, GAE, GCP, Fargate, etc. You push some buttons, they manage your cluster/services.
People still think they can get a free lunch by downloading some free software. If that were so, Windows and Mac would be dead and Linux would be the only desktop OS. But good news: I hear 2030 will be the year of the Linux Desktop!
Ahahahahaha, kids say the darndest things these days...
We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).
Far simpler if you can include the fact that with Kubernetes you can install new applications as simply as "helm install prometheus".
I wasn't aware they were offering fargate on EKS now. They weren't when I looked at it last.
My big issue with it is the underlying complexity and house of cards nature of running your control plane as sidecar containers on your runtime infrastructure.
You CAN set it up and run the control plane out of band, but last I looked there wasn't step by step documentation for doing so. I also couldn't find anyone doing it in prod which to me is a nonstarter. If I can't figure it out myself and can't hire for it I'm not doing it.
I'm sure it's fine on google cloud, but ECS solves 90% of our problems AND integrates with everything else we're using already.
I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so programmable. The idea that you'd need a completely new full-time team doesn't grok for me; that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers.
I do take seriously the other issues raised RE: security (though one GKE cluster per security boundary is a perfectly reasonable approach and gets you further than you might think).
> Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy
I think this is a different issue than the OP was raising; in any case, in order for a technology to succeed, you need to have subject matter experts embedded in your dev teams, or a separate function that provides the service to the teams that use it.
In the context of the OP, I think your case would be more like saying "hey data team, you need to build your data jobs into AMIs, go figure it out". Regardless of the technology chosen, it's not going to succeed if the teams doing the work don't know how to use the tools.
It’s great if their own solution works well for them at less cost, but that system didn’t built itself and has non-zero maintenance costs.
In that case I would expect Coinbase to write blog posts on how their setup is the absolute best solution to their problem, and not how they refrain from adopting the best solution to their problem because they claim they don't have anyone on the team that is able to pull that off.
The gotcha there is that it rarely goes that way unless you have a very clear direction from senior management, at least at big corps. In most cases, it's just another thing that gets added to the pile, and it's incredibly difficult to migrate entirely out of whatever the old solution was, so now you end up supporting both.
The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding. Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.
: The control points that k8s exposes are not simple "drop in your provider here" bits of integration. You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them, and more access control to implement and verify.
Volumes just get attached (using PersistentVolumeClaims), and automatically migrate to a new node of the original pod dies. Vs. having to do some sort of rsync between nodes to keep disks in sync.
Secrets get encrypted by k8s and mounted where needed. I would agree that RBAC is a bit tricky but I don’t think it’s harder than IAM provisioned with Terraform.
If you are not using a service mesh for your VMs then you don’t need one in k8s. (I don’t use one, and rolled TLS to the pod in less effort than it would take to maintain TLS to the VM). The reason you want a service mesh is to abstract TLS and retry mechanics from the application layer - i.e. make your service authors more productive. If you don’t use a service mesh then you are back to managing TLS per-service, which is where you are with VMs already.
There are definitely more services you _could_ run, but in my experience these are additive, I.e. they are extra work, but give you a productivity boost.
Anyway, YMMV and I haven’t operated a system as large as Coinbase, so I could be missing something. Interested in hearing others’ experiences though.
The problem with that assertion is that it does not make any sense at all. For instance, storage and config/secrets is already supported out-of-the-box with Kubernetes. Even so, complaining about storage with Kubernetes is like complaining about EBS or EFs or arguably S3 in AWS. And if you feel strongly about service meshes then you really aren't forced to use them.
> Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.
There really isn't. For example, if they go with managed Kubernetes solutions then the only thing they need to worry about is to actually design their deployments, which would be very strange if they couldn't pull off. That's a ramp-up project for an intern if the solution architecture is already in place.
> You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them
There really aren't. Kubernetes' equivalent to EBS is either setting up a volume or a persistent volume claim on a persistent volume. Just state how much memory you want and you're set.
This seems like an organizational problem to me. "DevOps walls" sounds like there is a "DevOps team" (a famous DevOps anti-pattern) and there are knowledge silos between development and ops, which ironically is the exact opposite of what DevOps is about. What this also means is that developers need to be aware of the environment in which their services run and should be very familiar with how k8s works, why not take that as an opportunity to learn?
The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it.
That being said, many companies don't need any of it in the first place for their scale, and that's probably the biggest issue with K8S today.
Once you have a managed cluster, the deploying apps is fairly easy. A single container/pod is a 1-liner and you can work your way up from there.
What do you find most painful about your use of bare-metal servers? The thing that I like most about a hyperscale cloud provider is the level of redundancy, including even multiple data centers per region, and their built-in health checks and recovery (e.g. through auto-scaling groups) based on that redundancy. With bare-metal servers, I'd have to cobble together my own failover system for the occasional time when one of those servers goes down or becomes unreachable due to a network issue. And of course, I'd probably find out that my home-made failover system doesn't actually work at the worst possible time.
That stack is super complex and utterly bespoke to the company.
With k8s it’s standardised and usually better quality.
I would really appreciate a more mature ecosystem.
Yes. I helped push out a k8s platform where I work, and it's been running well in production for ~3 years.
We most definitely did not start with "we want to deploy k8s". We started with "we're being asked to meet certain business requirements", which lead to "we will need to change how we do some of our development and deployment", which lead to "our platform will require these characteristics."
The easiest way to get those turned out to be buying a packaged k8s from a vendor (OpenShift, not that it matters; there's plenty of options).
Most of the pain with k8s seems to come down to people wanting to polish their CVs (by "having done k8s"), or people who sneer at packaged/hosted solutions because they want to build a cottage industry of building k8s services from scratch because it's their idea of a good time.
Unsurprisingly both take orgs down the route of pain, money, and regrets. Oftentimes the people driving this decision then prefer to say the problems is k8s, or containers. Or microservices or whatever fad it was they were chasing without understanding whether it would be a good idea.
Being reluctant to adopt new technology unless you really really need might be a more sensible thing to do.
Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on digitalocean. If I need more power for my cluster, I increase the nodes through digitalocean and k8s automatically moves my containers around how it deems fit.
I used normal blue/green deployments on self-managed VMs in the past, then worked with beanstalk, heroku, appengine and I much prefer k8s. Yes it's easier on heroku, but try to run 2-3 different containers on the same dyno for dev to keep cost down. On k8s I can run my entire stack on one single small digitalocean $10 VM if I wanted to.
I wouldn't even know what I else could pick that gives me equal flexibility and power?
K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said.
Also, we run a VPC-only K3s node for some simple internal tools that works great as well.
We do exactly the same thing! We have a one-node k8s for all these dev things that just works. Everything is containerized for local dev anyway so moving it to k8s was just writing the deployment manifest.
On heroku, all of these would be separate dynos (or one glued-together dyno that does everything). On a self-hosted VM we'd have to deal with managing that.
I liked this approach so much that I now have a small 1-node personal cluster that hosts all of my private hobby projects that aren't ready for prime time yet, that were on heroku previously. Costs me only $10 + (persistent storage + IP address if needed)
Sure, maybe you two (@dvcrn and @arcticfox) don’t work together and don’t know each other, but it’s definitely more entertaining imagining the scenario above.
It seems Dokku wins at "easier" since in many cases, you can just push the application code you used for development and the required stack is automatically detected. Adding a database is two commands. No need to know Dockerfiles.
I'll look more into it, but it didn't really inspire confidence.
k8s has a loooot of stuff it can do and reading too many blog posts that go into too much detail can be intimidating, so my advice would be:
1. Create a dummy cluster on digitalocean (or docker desktop with k8s support / minikube), then setup kubectl to connect to it.
2. Start with the difference between resource types: What are deployments, what are pods?
3. Create a deployment manifest that just tries to pull a container from some registry, apply it with kubectl -f foo.yml
4. Play with kubectl to inspect things: kubectl get pods, kubectl get deployments, kubectl describe pod xxxxxx, kubectl logs xxxx
5. Learn how kubernetes ties things together through labels: Create a ClusterIP/LoadBalancer service and try to get it to balance to your pods from above through labels (https://www.digitalocean.com/docs/kubernetes/how-to/add-load...)
Deployments/services/(pods) are all you need in the beginning for running containers on k8s and exposing them. Of course then there are things like persistent storage but if your app is made to run ephemeral, you likely have storage/db setup externally already.
For running through the CI, once you have your manifests you could run kubectl apply directly through the CI if you wanted to. We are using terraform in front of k8s with hashicorps hosted state, then run `terraform plan`. If it passes, on dev we automatically apply to the cluster, on prod there is a manual step through the hashicorp admin UI that needs an apply trigger. Then there are more advanced tools like spinnaker that can be used to setup more complex pipelines on what to do on push.
without k8s some things would be hard.
distributed jobs? same; nothing prevents from spawning runners with adhoc libraries and queues.
managed infra? not specific to k8s
Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?
If you only need certain parts of what k8s offers, building those parts yourself can offer you more stability, control, and insight into what your application is doing.
As with anything else it varies case-by-case.
Not that there aren't good reasons or good outcomes from having done a deep dive, but just putting it there.
Building it yourself doesn't mean building all of it. For example, it's quite easy to get zero downtime deployments with a tiny bit of systemd configuration and the SO_REUSEPORT socket option. That seems easier for a team to understand than "here is kubernetes and everything that comes with it"
You need to deliver the application to the machines somehow. You might need to configure the network to reach the application, etc. All quite common tasks.
(And honestly, systemd can bite you just as much if not worse than k8s definitions, because the API is much less cohesive and the defaults like to cut off your hands)
The problem with that logic is everyone on your team will need to do it, so you're going to be stuck picking a standard. Should it be yours, should it be mine, or should we both just learn something with a large community behind it?
Nothing is perfect of course, but k8s makes a really good target for CI/CD, which is something you want when you're developing as part of a team. If you're not quite a team yet and you don't know how to bootstrap k8s and CI/CD, then you need to figure out when those types of things are important.
Probably lots of people could stick with a monolith and a VM for longer than they did, but automated testing will save you a fair bit of time if you're not figuring out how to do it at the same time.
Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. I personally would use the word "industry standard" a little bit differently.
k8s is based off of Borg, which has existed for far longer.
There’s really nothing like the full suite that kubernetes provides.
You know tech has made it when people offer argumentum ad Java.
'Sure, X might not be the best solution, but you can always buy someone to do it.'
k8s also adresses a very small portion of the market. If you have to scale, yes, you might need k8s. Chances are, you don't. Really.
I stopped counting the (supposedly big) customers that burnt themselves into k8s when they logistically need not to, and only brought organisational issues on them.
Setting up and taking care of k3s is much easier.
On dev I don't need a 10 node cluster if I had 10 containers running. One $10 vm is fine.
On prod I can start with a 3-node cluster for these 10 containers, then scale it up depending on traffic and needs while controlling my spending.
Not everyone has thousands of dollars of VC money to throw at hosting
I like loosely coupling things, and Kubernetes is the first ecosystem where that has worked well for me. (OK, it worked great when I worked at Google, but a lot of effort was put into that by thousands of people.) For example, for the first time in my life, I automatically renewed a TLS certificate for my various personal projects. When I started using Let's Encrypt, I just manually ran certbot every 3 months when I got a warning email that my cert was expiring. That is fine, but it's kind of a waste of time. There are tightly coupled solutions to this problem, but they basically require you to totally commit to their approach (Caddy is a good example of this). I use Envoy, but Kubernetes let me not care. I run cert-manager, which just runs in the background and updates my certs when they need to be updated. It's stored as a Kubernetes secret, which can be mounted into my Pod as files. When the secret changes, the filesystem is atomically updated. Envoy can notice this and start using the new certificate. cert-manager doesn't know anything about Envoy and Envoy doesn't know anything about Let's Encrypt. So I'm not locked into any particular decision -- I can change my CA, and nothing about my frontend proxy has to change. I can change my frontend proxy, and nothing about my certificate management has to change. This, to me, is a big deal. I have one less thing to worry about, and I am not locked into any other decisions.
I also like the flexibility with which I can write programs to manage my infrastructure. All the primitives available to me as a programmer are high-level and well-tested, and are the same things that the CLI tools do. For example, in preparation for HTTP/3, I needed some way to get UDP traffic into my cluster. My cloud provider doesn't provide a load balancer for UDP, so instead I wrote a program that watches changes to Nodes from the Kubernetes API server, and updates a DNS record with the external IP addresses of all the healthy nodes. Then I can instruct browsers capable of HTTP/3 to use that DNS address to attempt an upgrade to HTTP/3, and it doesn't matter that my cloud provider can't do that at a lower layer in the stack. The alternative to this approach is to basically commit to having a certain IP address available, and keep that updated manually. It's fine, but again, one more thing to worry about. I can take this exact code, and it will work perfectly on any other Kubernetes provider -- so I'm not tied to DigitalOcean, and I'm not tied to any manual processes. One less thing to worry about.
I agree that a lot of people get into a situation where they have to move hundreds of apps and tens of nodes all at once, and under those circumstances, it sure is a lot of work to figure out Kubernetes compared to putting a band-aid on the problem and getting back to work. The biggest problem is that you are probably facing some sort of crisis, and have to decide, with very little experience, whether you want to use a managed offering or build it yourself. Building it yourself is quite complicated. What CNI plugin are you going to use (they all seem both wonderful and horrible on paper)? Why do you have to buy five nodes only dedicated to master tasks, like etcd? How are we going to upgrade to the next version with no downtime? You can go managed, but then you give up a lot of control. Who controls DNS at the node level (fun fact: container pulls don't go through the same DNS stack that the Pod will eventually use)? How can you use gVisor to isolate pods from the host kernel? (You can't! You will have to run it yourself.) Compromise fatigue is going to kill you here -- you have a crisis, and all the options are bad. (I've been there myself. I started using Kubernetes because our Convox Rack was so outdated that we couldn't deploy new software anymore. We tried upgrading things, but it broke things even more. So until we got k8s working, and converted every workload from a proprietary format, we couldn't deploy software. It was frustrating. But the reality is that I wanted to switch a long time ago, so the transition was quite smooth, with no prior real-world experience. And now this problem won't happen again, because tens of thousands of people know how to deal with Kubernetes.)
I also agree with this article that Amazon's managed Kubernetes offering is terrible. EKS was my first Kubernetes experience, and it was clear to me that Jeff Bezos walked into someone's office and said "we need Kuberthingie in two weeks or you're all fired." The team saved their jobs, but that's about it. It's very much the managed Kubernetes solution for people that are locked into AWS already. What people really want is not Managed Kubernetes but "namespace as a service". They just want to kubectl apply something and let a background task provision their machines. They don't want to screw around with RBAC, service meshes, managing the Linux distribution on their worker nodes, managing the master nodes, etc. That service unfortunately doesn't exist. Maybe send me an email if you want to work on something like this, though, because I certainly do ;)
In summary, I get the pain points, but I think they are worth embracing. Things aren't perfect, but you are going to have pain points at all the big breakpoints in infrastructure. Going from 0 applications to 1 application is going to be a major change for your team/company. Going from 1 application to 2 applications is also going to be a major change, but most people overcome this with sheer willpower and tedium until they hit something like 10 or 15 applications, and then are up a creek without a paddle. I recommend embracing future growth early, so that your second application is as easy to run as your first. It's not hard, it's not time consuming, it's just very different from "I'll pop in a Debian CD and rsync our app over."
I think you just described AWS Fargate and Google Cloud Run.
Cloud Run just seems like standard "serverless" stuff, nothing to do with Kubernetes. (The downsides involve not being able to run applications that are designed to run on a generic Linux box; everything has to be specially developed. That is fine, and they have open-sourced all the tools necessary to move off of them so you aren't locked in, but it's a bigger paradigm shift.)
So you already spend about a decade learning all the skills. What the other guy is talking about coming from dev not from ops. If you come from dev you don't necessarily know what an ingress or egress is, and might never have done a blue/green deployment etc. This is all stuff that needs to be learned first. I worked with many many teams who had zero skills in data center tech before they were moved to k8s full time.
I personally like it to learn all that stuff. And I love that my job requires it now. But it's more like vim than like node-red, and that was a shock for many people, from engineer to EVP.
I think this is roughly similar to saying "if you want linux, you need people that know how to maintain it." Which is to say, you can create an architecture where this is absolutely true, but it doesn't need to be true.
The big issue with K8S right now isn't K8S; its that there aren't (big, well established) solutions like Heroku or Zeit or whatever, for K8S, where you don't need to worry about "the cluster", just like those solutions don't make you worry about Linux. K8S really is two parts; the API and the Cluster. The API is the more valuable of the two.
And, you know, maybe it won't ever get there. Heroku and Zeit solve strikingly similar problems to K8S. Maybe K8S just is a platform like that, but for enterprises who want to home-grow, and maybe most companies shouldn't worry about it. But I think the platform, and thus the community, simply needs more time to figure out where it makes sense.
Most companies shouldn't touch K8S. You'll probably regret it. But, to your second point: AWS literally has nothing beyond EKS/ECS + Fargate which approaches a Heroku-like service. Beanstalk is supposed to be that, but its really just a layer on-top of EC2 which doesn't touch the "ultra low maintenance" of a Heroku, or Zeit, or App Engine. So if you're on AWS, and you want to use their other excellent managed services, you either go outside AWS, or you'll go EKS, or you'll end up trying to in-house something even worse.
I think you’re right though - Kubernetes is a massive red herring, we should ideally be running containers/functions on as close to bare metal as possible. Fundamentally, VMs are the wrong abstraction if all your code is containerized.
Joyent’s triton is the closest thing we have to this... I really don’t know why AWS/Azure/GCP haven’t cottened onto this, it would massively reduce their COGS and improve our developer experience.
Because there are various "tiers" of users, some companies (like coinbase) could actually leverage K8s in their Codeflow/Odin project and prevent a lock-in. But a regular developer looking to just "get things shipped" isn't meant to waste his/her time with pure K8s.
> Kubernetes is a massive red herring
We agree but on a different note. The biggest selling point of K8s is it's API design. The entire industry needs to converge on one "defacto" standard of packing and deployment. Google's Cloud function is a perfect example of this. The API is based on K8s and Knative but under the hood it actually runs directly on GCE rather than GKE. What happens underneath is hidden from business developer, you only care about the data in your yaml and your docker image.
Conflict of interest. If k8s yields to the most revenue why would they try to decrease that? If some customers are so delusional that they go for an inefficient abstraction so be it. Btw. this is my experience with k8s too, people use it because it is a trend. Not a single company / developer could justify using it to me over leaner resources like EC2, ASG, cloud native resources, etc.
I don't know. Are you saying that staying on-prem means you have to use k8s? Google is not a single vendor?
>> Apple is already getting rid of their Mesos based PaaS and moving to Kubernetes.
And? Should every other company follow Apple?
https://news.ycombinator.com/item?id=22508968
I have to say, certain comments -- which I'm sure are just as real-world as yours -- lie in...let's call it tension...with the threads and comments here in this post. That everything in the old post appears to be subsumed into a "sysadmin II: internet boogaloo" of orchestrator expertise is humorously ironic.
If you can get away with it, vanilla docker hosts running docker compose provide most of the same benefits with a fraction of the cost. For most startups, that's a great way to avoid getting sucked into a black hole of non value adding devops activity. You lose some flexibility but vanilla ubuntu hosts with vanilla docker installs are easy to setup and manage. We used packer and ansible to build amis for this a few years ago with some very minimal scripts for container deploys.
Sorry I do not understand that statement, in my naive opinion Docker Swarm seems to be a thing. Care to elaborate, please?
We are actually currently in the process of migrating from Docker Swarm to k8s and I am not 100% sure that's a good idea. We will see.
https://www.mirantis.com/blog/mirantis-will-continue-to-supp...
To be clear, I use Docker Swarm in my home cluster due to simplicity and ease of use. Unfortunately that pattern hasn't scaled to the Enterprise.
They never really retired it but at this point it's a footnote in Docker releases.
I've not actually encountered it in the wild in four years or so and never in a production setup.
It's just so damn simple in comparison to k8s - basically, if you know Docker Compose, you know Docker Swarm.
I appreciate it doesn't have the full power of k8s, but it has what most apps need: simple deployments, zero-downtime updates, distributed configs and secrets.
Seems like EKS is great for people to start looking at AWS without vendor lock-in, but then people switch over to ECS and/or Lambdas for compute.
Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.
I mean, k8s has 1000s of features. I am not using every one of them. I carved out the subset I need, and it is FANTASTIC. I get
* Great rolling deploys
* Self healing clusters
* Resource sharing / bin packing (rather than have 20 half used servers, I can have 6 much more heavily utilized servers)
I could maybe frankenstein these features on top of something else.. but frankly it seems absurd. If these are the only 3 features I use to run docker pods, k8s is a HUGE win to me.
there is nothing special about K8s in your situation.
ECS does not bin pack, does it?
Not available for those of us who choose not to AWS.
Serverless seems better and cheaper (lambda, GC Functions, Azure Functions). Pay as you go with no overhead costs. In many cases usage may never exceed the free credits.
For the database and persistent storage, you need a cloud native service like S3 and Postgres RDS. (Running a database in a container is asking for complete data loss from what I understand.) This is the primary cost for my tech stack and for lite loads $15 is about as minimal as possible.
So I don’t see how using containers could be cheaper unless you are keeping the data in the container, which is a bad idea since containers are supposed to be stateless.
So serverless cloud native tech seems better by far from everything I’ve experienced.
Also when using a tool like serverless framework, deploying an entire stack of serverless resources is dead simple.
I’ve tried a few times to see the value of containers, but I just don’t get it.
In my experience, multi-hour control-plane outages are very rare, and I've only had a single data-plane outage (in this case IIRC it was a GKE networking issue). Worst case I see is once or twice a year I'll get a node-level issue, of the level of severity where a single node will become unhealthy and the healthchecks don't catch it; most common side-effect is this can block pods from starting. These issues never translate into actual downtime, because your workloads are (if you're doing it right) spread across multiple nodes.
I wouldn't be surprised if EKS is bad, they are three years behind GKE (GA released in 2018 vs 2015). EKS is a young service in the grand scheme of things, GKE is approaching "boring technology" at 5 yrs old.
I'd be surprised if they have the same operational maturity, which translates to uptime, which was what I was talking about.
A quick search turns up these SLAs:
<bad link>
vs
https://cloud.google.com/blog/products/containers-kubernetes...
EKS: 3 nines
GKE: 3.5 nines
So Google is committing to 1/2 the downtime.
(Note the GKE SLA is for regional clusters, which is what you should be doing if you care about uptime. The zonal cluster SLA is 2.5 nines. I couldn't find a difference in EKS, maybe there's an equivalent better SLA for regional clusters I couldn't find.)
Edit - that EKS SLA link formatted weirdly, and so I did a little more digging and found a more recent SLA which matches GKE: https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-ek...
So, per my original comment, I am surprised. (Having never used EKS directly I have no idea what their actual uptime is; in my experience GKE has been way higher than 3.5 nines, but obviously I don't have enough data to make statistically significant observations on this.)
We haven't had any issues with k8's upgrades either.
That to me is a product they offer because someone else is offering it, but they don't want you to actually use it.
If you're actually looking to build isolation in AWS then you're going to need EC2 dedicated for your EKS member hosts. So you're not getting isolation for $200/month (I'd have to spec it out but dedicated hosts are pricey to the point that it'd be competing with physical hardware in a colo).
That $200/month for not-really-isolated is also per plane. So if you want a separate staging environment it's another $200/month. Client API sandbox? Same thing. It's wack.
And that is why pricing and features sit right at “good enough” and not great.
Introducing Kubernetes into our org allowed us to do this; we would have never gotten here with our legacy deployment and orchestration Frankenstein. The change has been so positive, that I adopted Kubernetes for my solo projects and I am having a blast.
I understand Coinbase's position, and they need to stick to what works for them. I just wanted to bring up a positive POV for a technology I am becoming a fan of.
For example a lot of people would find writing scripts cumbersome, but not a person who's written a lot of them. They're not any more fragile than other logic error capable software is
With k8s, a horizontal autoscaler is a few lines in a yaml file and the result works in any cluster run by any vendor.
No tool "works the rest out". It's _always_ a compromise, because inherent complexity can never be removed, only moved. What you gain in one area, encumbers another.
You may freely use k8s, but it's not magical nor easier to use than any existing systems. In fact adopting it often takes non-trivial time and the web is full of failure stories with very benign warnings and catastropic results.
Maybe I'm spoiled by GCP and it's not the same in other providers; but I can have a brand new debian VM configured and operational in less than 10s (less the time that I run my own startup script).
Debian machines spawn with incredible speed; not much slower (if at all slower) than a new container.
There are also many options for creating Debian based image files with all you need baked in.
I guess it depends very much on the application you have.
I think I've been in total of one company where it was somewhat true, and even there we didn't do it.
But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment.
But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC system with a strongly conservative development and operational tradition.
I understand loving efficiency and time savings but not everything simply works better with k8s.
Would you share your experience?
At my current job, I use Azure's managed Kubernetes service, which does a great job at providing a consistent environment that's very easily managed, no unexpected updates, great dataviz, and if you choose, simple integrations to their data storage solutions (run stateless K8 clusters if you can) and key vault. We don't do much outside of our kubectl YAML files, which as commented below has a de-facto understanding by a large number of people.
CVEs will always exist, which is why network security is important. I think we can agree that the only ingress into your cloud environments should be through API servers your team builds, and everything else should be locked down to be as strict as possible (e.g. VPNs and SSO). With a system like K8, so many eyes on the code mean so many more CVEs will exist, so I don't find this argument compelling.
My team, and so many other teams worldwide are betting that the K8 community will accelerate much faster than roll-your-own solutions, and K8 gives us the best opportunity to create cloud-agnostic architecture. Additionally, helm charts are easy to install, and afaict more software vendors are providing "official" versions - which means for a team like mine, which is happy to pay for services to manage state, in the same vain a company chooses AWS RDS over managing their own Postgres server, we can get the same benefits as the author with a cloud-agnostic solution.
Personally I've found Mesos much easier to manage, secure, and operate than k8s. However, when it first came out all the cool kids were using it, then most of them jumped ship to k8s. AirBnB's Chronos is now pretty much a dead project, Mesosphere's Marathon is now gimped (no UI) and major features moved into DCOS. At the same time, Mesosphere (now D2iQ?), now seems like it's more focused on k8s.
k8s is everything plus the kitchen sink, and managed k8s isn't the killer feature I thought it would be. I don't blame people for not jumping on the k8s bandwagon at all.
In addition to the well-known integration points (Container Runtime/Network/Storage Interfaces), there's things like the lack of a good built-in user authentication mechanism with Kubernetes, which means you pretty much always need some external authentication service for your clusters.
That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem.
Go spin up Keycloak, join it to your user-directory of choice (or not and just use the internal directory), configure it as your authentication provider, done.
Which has the advantage of not needing any external databases.
In many corporates you also now have the challenges of cross-team/department work, for the k8s team to work with the AD team to get it setup.
And still that won't get you away from the problem that without a first class user / group object in k8s people often end up running into problems with JML processes over time and mismatch between AuthN and AuthZ...
Nomad already does the container part, and Consul Connect is the networking overlay / service mesh. Work is being done to get Nomad better integrated with Connect.
Would love to hear what else you miss from swarm: schmichael@hashicorp.com
Update: Honestly if you haven't tried out our Consul Connect integration, please do. mTLS with just "connect { sidecar_proxy {} }" stanza and Consul running: https://www.nomadproject.io/docs/integrations/consul-connect...
The nice thing with Nomad is, thanks to it's straightforward design/approach, it should be easy(ish) in the future to swap it out and go with something else if we outgrow it (or HashiCorp abandon it for some reason).
That being said, it has been concerning to us to see the big institutional players move off of mesos and aurora. We wouldn’t like to be active maintainers, but we could.
I think that’s the main difference between Mesos and K8s: with K8s, we wouldn’t want to maintain it, and we wouldn’t be able to (since it’s so large). Somehow mesos and aurora feel more manageable.
Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure.
edit: meant to say mesos but applies to both
But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team.
In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.
Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement.
When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(.
The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora
I actually think managing aurora configs is way easier than managing yml files, and I agree that I think aurora configs were ahead of the game: having access to python in your config feels like a super power. I feel like we’ll converge on something that compiles aurora configs into yml files, prior to runtime.
That being said, we’ve never been able to get good editor support for things like “go to definition”, with the whole “include” syntax. We have maybe 2-3k aurora config files, of which maybe 100 are shared boilerplate. Do you have any advice on this? I tell vim to treat them like python files, but pylint hates them :)
We were bummed by the PMC decision too. I think some people at my company have considered becoming maintainers over the years, but, for the most part, everything “just works”, so we haven’t felt a selfish need to, so to speak. I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That being said, I’ll set aside some time this weekend to take a look at some issues. :)
Pystachio was indeed very forward looking and the folks who worked on this at Twitter at the time deserve all the credit there.
I think what you mention is a general problem I've encountered with IDEs when it comes to dealing with Python (esp. the "go to" issue you mention). Even when I've had to touch the Aurora client code, which is full on Python code, I've come out pulling my hair thanks to PyCharm acting wonky.
> We have maybe 2-3k aurora config files
Those are some big files! The boiler plate stuff is definitely something I've heard before from users but, unfortunately, there doesn't seem to be a better answer.
When it comes to managing job configs, I'm pretty low on the pecking order in terms of knowledge since we ended up creating our own thrift client using Go to use with Aurora. (As a consequence, all our job definitions exist as Go code.)
Stephan Erb (https://twitter.com/erbstephan/) may have better advice in this case. Some of the Twitter folks may have good info too, but they've been radio silent for months.
> I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers.
That's definitely a great point and a great compliment to the project. There's a lot of love that went into this project and I'd be ecstatic to get some new contributions, even if it something simple like fixing documentation or bumping up dependencies :D.
There's also https://discuss.hashicorp.com/ and we're discussing (harhar) how else to improve our community relations.
We use ECS for the majority of our container orchestration and love it.
I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.
<3 Nomad. However, Nomad only satisfies a really tiny part of Kubernetes ecosystem which is the ability to pack containers and schedule them efficiently in a cluster; Plus, it scales really well. Kubernetes provides a bit more than that.
Another thing that might have been just me, but I really couldn't see any structure in Mesos deployments. Yes, you could run X, Y, Z on top of it to get various features, but they all had separate APIs, separate input files, etc.
Having used k8s before that it was a huge blow, since at the time (late 2018) I was used to think that Mesos et al were more mature and advanced, and instead encountered k8s circa 1.2 with less cohesion :/
The article doesn’t claim that K8s is a bad solution. It only claims that migrating to K8s doesn’t work for them at this point in time.
If you are starting something new, or rebuilding a system entirely, and if you are on GCP, and you anticipate the need for scale, then K8s/GKE is a sane choice. What would be insane is trying to roll your own solution. There’s no avoiding the complexity of managing infrastructure, but at least with K8s you won’t be alone, and you won’t make the mistakes others have already made and fixed. Some people never seem to get K8s, in the same way some people never get functional programming. You might be one of them, so that’s something to take into account.
A lot of people have had bad experiences with K8s, but just as many have had bad experiences with Docker, or with Linux, or with computers in general. This doesn’t mean the technology is bad. It just means that not every good technology will work for you or your use-case. Kubernetes is a solid choice, especially GKE. But think carefully before deciding to use it. It is one of those tech decisions that will dominate the rest of your choices for years. In my case, it was a decision I‘ve never regretted.
- container orchestration platform is Odin + AWS ASGs (auto-scaling groups).
- Codeflow (our internal UI for deployments)
- Odin kicks off a step function and begins to deploy your application. New VMs are stood up in AWS and loaded into a new ASG, your software is fetched from various internal locations, a load balancer starts health-checking these new instances, and eventually traffic is cut over in a Blue/Green manner to the new hosts in the new ASG behind the load balancer.
- To handle secrets and configuration management we have built a dynamic configuration service that provides libraries to all internal customers with a p95 of 6m
- re-scheduling/moving of your containers if your VM dies/becomes unhealthy in your ASG
So if your company has all of that, I agree no reason to use Kubernetes. But what if your company doesn't have any of above system? Kubernetes.
I have used bare ec2 vm, deploy with Ansible, Capistrano, Chef, Docker..all of them. I even roll my own autoscaling with Consul, SQS(for termination notice) and I have more downtime than when using Kubernetes.
With Kubernetes, the learning curve is very high, but once you get it. It's painless to bring in new service, with AWS ACM to terminate TLS, and an ingress controller, pretty much as long as you have a Dockerfile, you can run it.
I found it interesting that for someone that doesn't use Kubernetes they spend a lot of time describing it.
There are benefits in using mainstream solutions like Kubernetes. I've spent over a decade building distributed systems from Hadoop to Mesos and Kubernetes and have seen the pains of datacenter and AWS, Azure and GCP and all I can say is GKE works great so I don't buy the simpler in-house argument. Once you start properly integrating concerns that need to be end-2-end integrated, the complexity explodes and you end up with a partially working system.
I do believe that serverless will likely make Kubernetes and similar-level tech irrelevant for most users, however only part of a rich ecosystem that provides all other concerns.
Eventually something akin to Ubuntu will grow up in the k8 ecosystem and people will stop complaining that WiFi doesn’t work.
I was very much looking forward to OpenShift/OKD to be "the Ubuntu" of K8s, but their targeted scope just keeps getting bigger (or it always has been big, not so sure anymore).
On the other end, K3s from Rancher is a minimalist distribution, and I think it'll limit its adoption in sophisticated environments with larger teams.
Does any other K8s distro look like a good candidate to standardize on in the future?
Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers
The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we have separate repo for sharing massive container images sometimes multiple GB rather that a simple file that can be tracked in git and will produce the same image every-time has been very eye opening.
Note: NixOS isn’t perfect but it has a growing community and hands down the best thing out there except for maybe guix which has container support too https://guix.gnu.org/blog/2018/tarballs-the-ultimate-contain...
I imaging a ci/cd flow where it build the image and installs your app via a local Nix package built from source in your git repo and all that needs to be shared is the git repo and you have everything you need to reproduce the container er running in production.
Nix packages are declarative and idempotent. Everything is based on a functional package manager and when you configure your container via nix you get the same thing every time and can easily mix and match other packages and dependencies.
A contrived example is you have a program that depends on multiple versions of python, because nix does not globally set dependencies rather links them locally you can have both with versions of python without any conflicts
Containers built from Dockerfiles are difficult to exactly reproduce too. A very common practice is to update the package manager in which depending on what the server returns will be different from one day to the next where as nix hash addresses and builds from source everything (you can use caches to speed this up) and you get the same thing every time.
Note: NixOS isn’t perfect in terms of being purely functional as many package builds use scripts in pearl,bash,etc but from my understanding these are usually source specific reproducibility issues with Building particular packages and generally speaking things works as expected.
If NixOS is only "generally speaking" correct because of the order it runs its scripts, then put it in a Docker container, where the whole point is to perfectly reproduce the order you run your scripts in order to minimize the errors that arise from doing otherwise. Still worried? That's why layers exist, so that every step you run you now have a static binary that you know will never shift under you, and that you can keep on building on top of.
In practice, building nix dependencies is reliable and reproducible because of the militant isolation of build environments, hash integrity checks on _all_ inputs, explicit dependencies, and the long tail of bug fixes related to non-determinism in unix tooling.
This is my personal experience, not abstract.
However, building the container from a configuration.nix file gives you exactly what you need in the container and nothing more. It's like basing your container from scratch in a Dockerfile and copying only the needed compiled dependencies and nothing more into the container.
> Containers built from Dockerfiles are difficult to exactly reproduce
This isn't true in my experience, at least for a level of exactness that could have an impact on the behavior of the software running in the container. I have been working with docker in various contexts for 5 years and have never experienced any issues related to reproducibility (which is pretty much the primary selling point of docker). To your point though, I can see how reproducibility issues could happen in theory, but since its not a problem in practice it doesn't make for a very compelling reason to try nix instead. Just to be clear, I'm not trying to disparage nix, it honestly sounds pretty interesting, especially as a way to automatically provision a local workstation (docker is not good at this), but I am still curious as to the practical tradeoffs in terms of nix vs docker as a tool for building containers.
Another advantage is a unified development environment as I can run the server in the container or directly on host machine in an isolated nix-shell.
Yet another advantage, is unlike the layer of a docker container, isolated dependencies are isolated and linked so packages link together based on a hash-address/path of a dependency in the nix store so there is no need to worry about order of steps but rather what depends on what making the cache misses in a nix-store vs a docker layer more efficient. For example if the first step in a Dockerfile needs to be changed all proceeding steps need to be rebuilt where as if a dependency in a nix file say paython-2 to python-3, every package that depends on python just get re-linked and only those packages need to be updated and you already have python-3 used in a nix-shell or in another container odds are it's already cached and ready to go!
"For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management."
So, with thousands (maybe more?) of "folks" out there, each with a different environment, requirement, background and so on. How can someone knows what most of them can or cannot do? Sure there is a lot of them posting about it in blogs and showing in conferences, but it still a very small subset of all the companies running containers today in the world, and the majority will not expose any info about it, for a myriad of reasons.
[1] https://www.reddit.com/r/CoinBase/comments/gh3b5t/coinbase_c...
[2] https://www.reddit.com/r/Bitcoin/comments/6gtwyi/every_singl...
[3] https://www.reddit.com/r/CryptoCurrency/comments/ggr80l/i_do...
In seriousness, we find k8s makes handling our traffic spikes a doddle. As in, we don't have to do anything.
Of course, we didn't necessarily need K8s to accomplish that, but it's the tool we used, and the complexity cost is paying off for us.
And then you're stuck with a proprietary solution that lags further and further behind established, open source projects.
Trading during high-volume hours was next to impossible: orders were not getting filled (sometimes they didn't even reach the order book!), the UI wasn't loading, etc.
Then they shut down for a while and rebuilt everything starting from the matching engine. I guess they also reworked their architecture because after that, it was smooth sailing ever since.
Funny, we're running it sanely without doing that. We've separated our clusters based on use-case - delivery vs. back-end, aiming towards the "cell-based" architecture.
> Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it makes them more difficult at this time)
...some details on the challenges they don't solve, or indeed make more difficult would be good.
But yep, K8s is complex. So, to paraphrase `import this`, you only want to use it when you have sufficiently complicated systems that the complexity is worth it.
At my current shop, we struggle to maintain k8s clusters with an 8 person team. We inherited the debt of a previous team that had deployed k8s and their old legacy stuff was full of dependency rot. We have new clusters, and we update them regularity, but it's taken nearly half a year so far and we don't have everything moved over.
You do need good teams to move fast; and good leaders to prioritize minimizing tech debt.
I think the way we've approached it achieves the same goal as just giving each team their own cluster to avoid them messing up other teams.
And yet, we keep reading about teams falling into the trap because their lead engineer wanted to put "production kubernetes" on his resume. I hope the k8s team adds a huge "Who is this NOT for" disclaimer to their docs (if it doesn't already exist).
The positive for me is I am learning a bunch of stuff during the process. The negative for the project is he expects developers with no previous skillset in this space to design all this new (for us) tech without introducing technical debt. The downside for the client is...well guess.
It's sad. I am rewriting a problematic legacy application and creating a whole new one. I guess I can put K8 and the kitchen sink on my resume when I look for a new job just before this thing implodes on release though. I just wanted to write clean code man, thats all. There is a sadistic part of me that wants to grab some popcorn and see how this explodes in his face...there is also a part of me that wants to be proved wrong...I hate this job.
I'm not against the microservice idea. I just rather focus on solving the problems the legacy system had. None of those had to do with scaling. They were related to a certain federal agency changing their mind every 2-4 years and really poor coding practices. A microservice with kafka and k8 don't really solve those issues.
Kubernetes supports mostly Linux, although has recently gained Windows node support.
But sometimes I wonder whether we are hurting our own careers not going down the k8s route. It really seems it has a bright future.
I believe the world will always have a demand for simple technology.
I think `rkt` and `lxc` are missing from your list.
https://learn.hashicorp.com/vault/kubernetes/k8s-reference-a...
https://github.com/sethvargo/vault-on-gke
In summary, for your security-critical workloads you're going to want to put them in their own cluster; treat k8s in this case as an API for updating the code that's running on your VMs. (Except your VMs can run a stripped-down read-only OS like Container-OS or CoreOS).
I'd take k8s over Beanstalk any time.
Kubernetes is not about scale, it is about defining primitives, everybody can use to describe their setup. It is a DSL everyone converges to and as a result of this unification products from different vendors can be packaged and deployed in the uniform way.
The article is clear: Coinbase has a very strong, tested and validated infrastructure and moving to Kubernetes 'just to be part of the hype' does not bring any benefits at this point. And a statement like that is a nightmare to some DevOps.
- Exchange that crashes during any time of high volume.
"Here is why, in our superior engineering judgment, you should do things the way we do in order to preserve uptime."
(To answer you question, probably because it uses the Medium code.)
Because the website is hosted on Medium.com and they make heavy use of lazy image loading.
It took ±8 secs to load for me, content was visible after 6 secs: https://i.imgur.com/6A09CDS.png
To be fair, I’ve visited websites that continue loading stuff for +3 minutes, Coinbase’s blog is lightweight in comparison.
>> at Google it isn’t uncommon for them to have multi-hour outages with GKE
For what it's worth we've been running multiple GKE clusters in production for over three years. We're medium size, with some dozen or so in-house services handling perhaps a total of 20k rps. We are rarely affected by any GCP issue, and as far as I can recall we have never been down due to a GKE-specific problem. In addition to the basic orchestration features we make significant use of ingress and storage primitives. It all quite literally just works.
Service discovery, cluster management, secret storage, these are all problems that we _already_ had. Containers (for us on mesos) just solved some parts of that picture.
I lived and breathed containers, distributed systems, config management, deployment pipelines etc. for years, and I forget that to many K8s is just seen as one magic bullet solution. You will have to pick it apart and interact with pieces of it if you really want to use it at a medium sized company. That takes a lot of research and understanding.
Coinbase provided an analysis worth studying. The major takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors when they just want VMs.
The bottom line is: K8s is awsome but it is complex. It is a Google-needs-oriented software. Sadly average project will not have that complexity you find in Google Plex.
I can run Oracle Database in 100MB of RAM...you need 4GB at least to run a K8s node... a costly option. Below ten servers I find no point in K8s: I highly suggest docker swarm.
Still, I honestly feel that we've taken a step back in the industry where we went from ec2 --> heroku --> k8s. But I know there are many people working hard to create the next infrastructure so we don't have to deal with containers and all that re-inventing dev-ops non-sense for every project.
K8s is complex because managing distributed services is. Not using it doesn't mean it goes away. The complexity migrates and ends up being bundled up in a separate tool or a runbook process or some script.
It's hard to maintain because the tools and apis are different from what some engineering teams are accustomed to using. Building an in-house tool gives them a warm fuzzy feeling and comfort that they can handle problems when they appear due to familiarity with their own code and design choices.
It's a fair trade off. I do wonder how much of the time spent doing this exercise could have been spent on K8S training.
I do feel that the K8S community do downplay how much a PITA k8s configuration can be and that the perceived robustness of cloud-managed K8S isn't up to scratch for something this complex.
I believe it initially shipped as part of FreeBSD 4.0 in March 2000.
Wikipedia says[2] it was in use in the late 90s in a hosting provider before inclusion in FreeBSD though. Interesting! TIL.
[1]: https://papers.freebsd.org/2000/phk-jails.files/sane2000-jai...
But it has a big learning curve and setup & maintenance are very costly. I don't understand why most orgs are moving to k8 considering this. When talking to my peers I often see numbers like 6 months to 2 years full migration with a very small added value - at least for 90%+ of the companies using it.
It usually boils down to attracting talents and keeping them excited trying the new shit.
K8s is simply a good default environment which provides rock solid stability for your applications by outsourcing the distributed systems complexity to your infrastructure team (whether its internal to your company or to a managed one like GKE). Teams are not using it just because it’s “cool” (maybe some are), there is no need to develop in house strategies to deploy and keep an app running and scale it (among other things; this is the lowest common denominator).
It’s the same reason why big data tech has somewhat standardized on a set of tech (spark, airflow etc): once people learn the system, they can focus on building products that provide value rather than building the products and the relevant infrastructure.
I wish someone can make a deployment orchestrator for your private DC that is as simple to use as Heroku is. Edit: fixed typo.
This is what I'm working on, although it is built on top of Kubernetes! We feel Kube suffers from a disconnect between the insane complexity requirements of enterprise deployments and what most coders and businesses actually need - which is exactly what you said: a consistent Heroku-ish experience on their own hardware or whatever cloud is convenient. Kube needs what Git needed - a GitHub. Feel free to reach out to me (email in profile), I'd be happy to chat your ear off about it!
I would rather just be able to do that than get into ECS/Fargate, much less K8s. It seems like all of that stuff is just adding more complexity for me.
Of course none of my projects are gigantic or need to be highly secure.
BUT, my prototyping turnaround is way much faster and more stable than K8s. Way less staffing too.
However the main point still stands that k8s is overkill for like 90% of deployments using it.
In the on prem case, you already have a dedicated infra team that probably has the tools to effectively deploy and manage a cluster.
Tools? Yes. Care and attention? Hell no.
The ops team at my work still has important services running on CentOS 6, and they are spending all their time trying to get Kubernetes configured and working.
You would be surprised at how much infrastructure out there is just glued together adhoc. Several senior engineers at my work are successfully blocking adoption of gitops and CD.
Even in their case, it looks like they took a conscious decision and documented it. I'd wish more teams would do that. You are free to use K8, and I'm sure right now it's still not a controversial choice that will run against walls.
Better go with Heroku equivalents than feeding a whole team, before you really need a whole team to feed Kubernetes.
Unfortunately most cases if I used Heroku I'd drive the cost way too high, as Heroku (or cloud) pricing doesn't change with local developer costs.
There are also application engines in major cloud providers, base on your technology. Some tech stack can run serverless. For self-host scenarios, things like Dokku could also work.
Cause in many places the cost of fully loaded experienced engineer might be cheaper than managed services.
At least if it's sane and well thought out it can be taken back apart and repurposed for something different. There will be cost. Consider it paying back the loan. That's why it's called technical debt. The terms of the debt looks good to me.
I informed them my team could do it all in under two days using Terraform + AWS + EBS. Unfortunately, they didn't take us up on the offer.
Sunk cost, etc.
I've never seen a company use k8s and not end up with a giant expensive team maintaining it all.
Fyi to anyone who is reaching (or close to reaching) this point: this is exactly the problem we're trying to solve with EnvKey[1] in a secure, robust, and scalable way.
Our current product runs on the cloud (using zero-trust end-to-end encryption) and solves the problem quite well imho for companies with up to 50 or so engineers and moderately complex infrastructure. It fits easily into either containerized or non-containerized stacks.
But our v2 is almost done after years of work, and I think it will now be able to handle almost any scale and workload. The launch target is August 1st. Some cool features it will offer:
- Source available self-hosting with auto-scaling, HA, and strong consistency that just works.
- "Config blocks" that can be used in multiple projects, allowing you to de-duplicate configuration and secrets.
- Version control with simple and advanced rollback capability.
- Comprehensive access logs with simple and advanced filtering and auditing capabilities.
- Ability to manage local environments.
- Ability to react to updates and e.g. restart servers when config values change.
- A CLI that will have full parity with the UI, either for automation or those who prefer a CLI-based workflow.
- An option to use our UI via an incognito web browser pointed at localhost instead of Electron (for those who hate Electron)
- Faster, lighter, modernized end-to-end encryption built on NaCl (v1 uses OpenPGP, which is great, but it's time to move on).
- Device based auth with optional passphrases (think SSH) and an easy, secure workflow for granting access to new devices.
- Ability to authenticate and invite users via Github, Gitlab, Google, Okta, or SAML (including inviting from multiple sources within a single org).
- Teams/groups and advanced group management: grant teams of users access to groups of apps, connect groups of blocks to groups of apps, etc.
- Customizable everything: environments, sub-environments, and access roles can all be molded to fit your workflow.
Our goal is to fully solve this crucial piece of the stack so that it "just works" with minimal time spent on integration (for new projects it can be installed and integrated in minutes). If you're interested (and/or want early access), submit your email to the form at the bottom of our site: https://www.envkey.com -- and of course if the v1 can solve your configuration and secrets management needs as it already does for hundreds of our customers, give that a shot! Upgrading from v1 to v2 will be very quick so you won't be duplicating any work.
Also, we're hiring--remote anywhere in the US. We'll put up a jobs page with more detail soon, but our stack is TypeScript (node + react), Go, and polyglot (since we need to write lots of integrations). Shoot me an email if this sounds like a system you'd be excited to work on: dane@envkey.com
If you don't mind me asking, how does the Go integration work? I initially thought it was actually some sort of alternate `os.Getenv()` that you imported, but that doesn't seem to be the case. And what would the latency be, for changes in environment variables being synced to running deployments?
When a process starts, envkeygo makes a request to our config service (via envkey-fetch) to fetch the encrypted config, then decrypts the config and sets them on the environment so they can be retrieved with `os.Getenv`. Both the lookup id (for fetching the encrypted config) and the encryption passphrase for decryption are initially passed in via an ENVKEY=... environment variable--you can think of it as a single environment variable that "expands" into all the others that you need.
Latency on a request is generally in the 150-300ms range from the US (our primary servers are in us-east-1).
EnvKey is strongly consistent and transactional, so once you make a change to your config, it will be available immediately for any subsequent requests. For now, it's still up to you to restart servers/services yourself after a change. With the v2, this will be scriptable.
We'll eventually hire outside the US too, but for now I'm trying to keep the timezone spread and administrative burden low :)
My personal experience is very good and i'm looking forward to a bright future.