Hacker News Reader

Comment by yongjik | Hacker News Reader

At this point, why not just drive it up to the logical conclusion? Treat your business model as cattle, not pets. Customers leaving? Fire up another business until capital runs out, and if it does, no worries, jut hop to another job!

Sorry, but I feel like I landed in crazy-land. Kubernetes is already an exercise in how many layers you can insert with nobody understanding the whole picture. Ostensibly, it's so that you can isolate those fucking jobs so that different teams can run different tasks in the same cluster without interfering with each other. Hence namespaces, services, resource requirements, port translation, autoscalers, and all those yaml files.

It boggles my mind that people look at something like Kubernetes and decide "You know what? We need more layers. On top of this."

View on HN

I'm in kind of the opposite position, as I feel confused about why I keep seeing so much resistance to higher levels of abstraction here.

> It boggles my mind that people look at something like Kubernetes and decide "You know what? We need more layers. On top of this."

I'm not involved with this project at all, but yes, I look at the massive Kubernetes deployments I've worked with, and conclude that toil and various other kinds of problems could be reduced with higher-level abstractions for declaratively managing configuration for all of these clusters.

If you wanted to run workloads on 100k servers, how would you do it? Would you have a team manually configure each cluster individually, or would you consider that there are a lot of similarities between clusters, and you only want differences that are chosen intentionally, and look into some abstraction to help keep the complexity down?

If 100k isn't enough for you, is there any number of nodes at which you'd consider building some tools support for your work?

What about data center count and change rate? There are a lot of business models that benefit from running workloads closer to customers, so it's useful to have deployments in many datacenters and clouds across the world, and for this to grow over time. Do you really not see why some people are interested in being able to bring up a cluster in a new DC by updating a few configs rather than starting from nothing by hand with kubeadm init?

I apologize if I've misrepresented your position by misunderstanding something. I feel like I'm missing or misunderstanding something from your perspective, but I don't know what it is.

More generally, I think it's extremely normal for people to build higher levels of abstraction as things grow. It's usually worth considering abstraction or architectural changes in a system every time it grows by about 10x.

Some of the most painful platforms I've supported in my career have been those that didn't do this, and instead tried to grow through brute force. Something can be fine to do once, troublesome to do 10 times, and overwhelming to do 100 times.

I think the comment you are replying to was objecting to the number of layers, rather than the level, of abstraction. More concretely, I'd guess they would not object to a tool that replaces k8s and offers a higher level api, but they do object to building said tool on top of k8s.

For myself, I think it depends on how leaky the abstractions are. The leakier, the fewer layers the house of cards can stand. That is, it's more about where you slice it than how many times.

That’s something that comes in time, if ever. If Kubernetes started by replacing the OS later of abstraction, say by requiring unikernel applications, then it would have virtually no adoption because we as an industry don’t know how to build and operate unikernels in production. You have to meet the market/industry where it’s at and then co-evolve with the industry into some tighter, more optimal form.

I mean, I agree in that I'd like such a tool too, but given that we don't have this hypothetical more-scalable kubernetes, how is it such an unimaginably shitty idea to build tooling for declarative configuration of clusters?

As you say, there's still a fundamental amount of complexity involved in operating a system, and the best abstractions limit the amount you need to care about in a given context.

Something I like about declarative configuration specifically is how well it helps you move information from ephemeral human memories and habits into something more-reliable.

Most places I've worked usually had some weird ephemeral things, oral history about what needs to be treated differently, many systems that needed conversations to understand, etc. The more you limit your use of global mutable state (interactively changing things in production), the less room there is for important stuff to live in people's heads.

When I can spin up on a new service just by learning how a tool works and reading their configs, then I have all of the "what" and "how" and "where" there to look up at any time in a consistent way. There's much less room for weird state or needing to do rituals or broken staircases or whatever. When I talk with people, I can focus much more on the "why" questions.

I might be more sensitive to this than some people, because I've got a shitty memory, but the more I'm able to just check a thing the better. Theoretically documentation can fill a lot of the same role, but I've never worked anywhere with consistently up-to-date and comprehensive documentation for what's deployed. When it's the only way to deploy at all, it's no longer optional.

It feels like the same sort of thing as a good static type system to me. You can do local reasoning with just what you see, and limit unexpected action at a distance. The types are mandatory, and machine-checked, so you can't get them wrong in certain ways, and you can't just skip writing them like you do sometimes for unit tests that would cover similar verification otherwise.

> If you wanted to run workloads on 100k servers...

But... Most don't run 100k servers. Most don't run 10k servers. I was working with a very large consumer insurance company recently and while they are really large for their segment, they're well under 15k servers in production. And containers aren't much of a thing. There's a lot of overengineered architecture going into not-so-hyperscaler/FAANG production environments compared to what Kubernetes' original use case was designed to solve for. The common consequence appears to be increased complexity and less operational system knowledge .

I don't follow where you're trying to go here, or how you intend this to contradict anything I said.

You're right that most people don't run 100k servers. In fact, most people don't run any servers at all, and don't use kubernetes. Some people do, and some of those people use approaches like this.

The comment I was responding to, by my reading, seems to express incredulity that anyone would do this, and seems to imply that this is a bad, counterproductive idea that nobody should ever implement. The most notable quote is "I feel like I landed in crazy-land". Do you read it differently from me?

My intent in my reply was to describe some of the use-cases where this is helpful. It's not only FAANG that run more than a small handful of clusters; there are plenty of smaller businesses that need to run large numbers of clusters.

I don't know anything about the company that wrote this article, but I found https://zitadel.ch/usecases on their website, and given that description, it sounds quite plausible that they run many small clusters in many clouds and datacenters. They probably don't need a ton of compute, but I wouldn't be surprised if they had latency needs for being close to their customers, or other specialized requirements that motivate dedicated clusters.

I also don't know anything about the insurance industry, so I'm curious to hear if I've guessed incorrectly, but it doesn't seem like the kind of business with especially high compute needs. I believe you when you say that your company has no use for this.

Can you believe me when I say that there are genuine problems these systems are trying to address? I keep hearing about companies who fund their employees building unnecessary pointless over-engineered production automation, but I have yet to ever encounter one in real life. Every SRE job I've had so far has been on a team that really wanted to invest time in improving infrastructure and automation, but couldn't get time away from the toil for it. If anyone can recommend a company that over-invests in infrastructure automation and architecture instead of under-invests, I'd dearly love to try it and see if it's as bad as I've heard.

Even without high compute needs or large numbers of clusters, there's still a lot of benefits you can get when you can move things to declarative configurations and away from global mutable state and human minds. I agree that this can go wrong, but that doesn't mean it's categorically worthless to try.

On the other hand, if you're just looking to gripe about it having gone wrong for you, could you share some war stories? I may be feeling idealistic from frustration with low investment in tools support and production automation lately, so maybe I could use some horror stories of it going badly to scare me straight. :)

I think that the contradiction here comes from the fact that these tools that are suited for large scale operations, like Kubernetes, end up getting standartized and adopted even by smaller corporations, which have neither the specialists, nor the resources to utilize them properly. Be it because of FOMO (fear of missing out), CV driven development or something else entirely, but i've seen this a number of times in the industry and it's always gone poorly. Instead of relatively quick deployments with Docker Swarm, Hashicorp Nomad, Docker Compose or anything of the sort, it suddenly becomes an uphill battle of trying to administer the darn cluster, as opposed to just being able to develop software, even with turnkey solutions like Rancher ( https://rancher.com/ which is great, by the way), especially if the company has only recently adopted Kubernetes.

In contrast, i think that Docker Swarm does a much better job at smaller scales, because:

  - it uses way less resources than Kubernetes (which matters on smaller nodes)
  - it is included in the install of Docker and therefore is easy to launch
  - it supports the Compose format, which is often used for development with Docker Compose
  - it is relatively simple, yet handles most of the concerns of smaller scale projects (i.e. excluding things like autoscaling or CRDs)
  - there are solutions to manage it with web interfaces as well, like Portainer, if necessary ( https://www.portainer.io/ )

Or, if you need Kubernetes in a non-managed format, i'd suggest that anyone take a look at projects like K3s ( https://k3s.io/ ), which attempt to cut out all of the unnecessary components and features of Kubernetes, that most people won't use.

In my opinion, going with Kubernetes as a whole or even full Kubernetes distros (or building your own clusters for prod) is akin to choosing Apache Kafka because you want to build an event driven system, and then needing to throw a team at the thing to actually keep it running smoothly, instead of just choosing something simpler, like RabbitMQ or ZeroMQ.

I'd argue that most companies out there don't even have "SRE jobs", but instead it's a duty that gets thrown upon regular devs who also need to deliver features. It's probably easy to overestimate how capable most companies out there are in regards to throwing resources at problems to make them disappear, mostly due to a lack of said resources.

Startups are insisting on K8s experience despite targeting a market that will never need more than a single server if they write the application properly.

I look at "if you've got 100K servers..." and ask "why does anyone needs 100K servers?", rather than "how would I manage that?".

Each layer adds complexity and reduces performance. Less layers are better. Write Unikernel applications

> At this point, why not just drive it up to the logical conclusion? At this point, why not just drive it up to the logical conclusion? Treat your business model as cattle, not pets.

We're already there. Cue "We invest in teams not ideas"[1,2] and "We're pivoting since we couldn't get a product-market match, but have millions in the bank"[3]

1.https://www.forbes.com/sites/ilyapozin/2015/10/08/4-reasons-...

2. https://www.gsb.stanford.edu/insights/do-funders-care-more-a...

3. https://triplebyte.com/blog/rethinking-triplebyte

> It boggles my mind that people look at something like Kubernetes and decide "You know what? We need more layers. On top of this."

This statement is less about adding another layer. It's more about not falling in to the same trap that we did with VMs in 2010--managing them by hand, producing unreproducible configurations, "tending" to them. The same pattern is happening with Kubernetes, but it fundamentally runs on computers at the end of the day so it's prone to similar types of failures.

"You know what? We need more layers. On top of this."

Heh. Service Mesh!

K8s on K8s! K64s!

But can it run on an N64?

it starts at Nintendo 69.

Kuberneteuberneteuberneteuberneteuberneteuberneteuberneteubernetes is a perfect name for the beautiful system you are about to architect: it's an ancient Greek word for helmsman-man-man-man-man-man-man-man, who is in charge of helmsman-man-man-man-man-man-man, who is in charge of ...

(...sorry =_=)

You joke, but have you looked at K8s VirtualCluster? Separate control planes for each tenant within a K8s cluster.

https://github.com/kubernetes-sigs/cluster-api-provider-nest...

of course. of course!

it’s k8s clusters all the way down

"Nobody will ever need more than K64." - Robo-Gates

May I introduce you to https://gardener.cloud?

A service mesh isn't a layer on top of K8s, but rather fills gaps in K8s feature set. Service meshes are great for managing traffic between services: authentication, authorization, encryption, traffic shaping, etc.

It's fine to like them, but it's hard to say a separate control plane, proxies, etc isn't another layer.

It's a layer you need regardless, it's not unique to Kubernetes.

I mean, depending on the business, employees aren't trusted to understand the whole picture regardless. Many employees at traditional business don't even have administrator access on their laptops, depending on their position so it's not logically inconsistent with how things seem to operate. With that lack of big picture overview, it makes things hard to scrutinise since you're only seeing one sliver of an implementation (ie; "It saves money" without seeing, let alone understanding the technical requirements and vice versa)

Limited admin practice isn’t just to save money. It’s a security practice and it’s a good one. In a large business, no one understand the whole business maybe except legal department and CEO.

> In a large business, no one understand the whole business maybe except legal department and CEO.

lol... what?

A CEO of a 20,000+ person company literally sees projects as revenue sources and a deadline, nothing more.

To get it delivered, he/she walks down the chain of managers until they gets answers. It's might as well be a black hole.

I only work for three large (2000+) companies. I’ve written reports to executive management committees. CEO never really cares about projects per se. they care about business units. Each business unit can have important projects. CEOs and management committees don’t really dive into them, they care about how each business unit are operating according to his management goals. CEOs aren’t gonna care about Python or Golang.

> Limited admin practice isn’t just to save money. It’s a security practice and it’s a good one.

And it's a red flag. Means that IT is running the show, not engineering.

I agree in principle! To be a bit clearer, I was thinking more in terms of locality so for example, "Users don't even have access to their laptops right in front of them" let alone visibility into their own department or even wider organisation due to access policy restrictions. Being able to write or edit everywhere would be ridiculous but sometimes there are (admittedly intangible) upsides to transparency (read only access in a technical sense) to most aspects of the wider business

You think the _legal department_ understands the whole business better than anyone? Where have you been working?

Every company I work for, legal usually sign of every deal, in terms of legality of the contract. They know every contracts/clients we have. They know where all our business is. They might not care whether our business plan is or works, but they know what our business is. Same can be say about finance group.

Also, the caliber of engineers at Google might not be the same as RandomCo where admin access isn't allowed for engineers.

> Fire up another business until capital runs out, and if it does, no worries, jut hop to another job!

I hear people say Google runs their business this way. So perhaps it makes sense that Kubernetes came from them.

> Ostensibly, it's so that you can isolate those fucking jobs so that different teams can run different tasks in the same cluster without interfering with each other.

I think the point is so the teams can run services at all. Isolation isn’t the point, it’s breaking a dependency on an ops team (Kubernetes in a sense is ops as a service).

The only explanation I have that makes any sense is that it is a very Application Developer solution to a System Administration that hasn't matured it's abstractions and is now trapped in Tech Debt purgatory.

With a sufficiently-large N, N+1 is barely noticeable.

Reply on news.ycombinator.com