Hacker News Reader

Comment by nvarsj | Hacker News Reader

I came here to post pretty much the same thing.

Networking in k8s is the poorest designed aspect of Kubernetes. So much so, I'm honestly surprised it didn't kill off K8s early on. You still can't get proper service layer load balancing out of the box w/ iptables.

I'm not sure the history - it's the simplest thing possible, I guess, but doesn't perform very well at the most basic of tasks. To try and solve this, we have the current mess of service meshes, CNI, etc. It's going back to the middleware days of yore - which I think anyone with operational experience should know we really need to avoid. Having multiple layers of network proxies and masquerading between service calls, on an internal network, is just ridiculous and difficult to debug or operate at scale.

View on HN

K8s networking is just typical networking. Networking always has to be made as complex and clunky as possible. Obvious solutions like unified IPv6 address space must be avoided as they would simplify the system and avoid the need for NAT.

I’m being snarky but it is a long time observation of mine. When I see a network config I am more often than not shocked by its needless complexity.

Folks who say stuff like this aren’t taking into account the number of real world constraints on networking at a large scale. There are bandwidth, reliability, security, management, cost, and performance considerations behind the complexity, just as there are at other layers of the stack. Folks higher up the stack want a simpler abstraction, but you can’t short-circuit physical limitations with a unified IP space.

To take the first property you mention, as an example: How do you feel bandwidth is helped by the proxying / NAT approach compared to traditional IP routing and Internet addressing?

Caching proxies and traffic controlling NAT.

Why do you need NAT to have caching or traffic control?

"K8s networking is just typical networking."

K8 networking is far more complex than typical networking and with less visibility

That is not fair, kubernetes uses the same existing technology in a well integrated fashion, including firewalling. That is what Network Policy and Service objects gives you.

Communication between pods running across different computers gets encapsulated in another IP packet. This is pretty standard across the industry.

To your point, which “k8s networking”? My beef with k8s is that it punts on networking altogether and you have to cobble your own with some CNI or another, and depending on which you choose it will significantly impact upstack choices such as load balancing and ingress as well as downstack choices, such as whether you can use Tailscale to connect multiple nodes together (the CNI and load balancer must be compatible with). At least this has been my experience as someone who isn’t a network specialist.

What rescued kubernetes was adoption by AWS and Google. If you want to host Kubernetes yourself it is still a waste of time broadly speaking.

Kubernetes didn't need to be rescued in the first place. If you know how to manage any Linux box, setting up kubernetes is as easy as "# apt install kubelet kubectl".

Broadly speaking kubernetes saves time by providing an api for existing Linux technology. For example, iptables, ipvs, storage, self-healing, virtual ips, etc via yaml.

Kubernetes was created to give anyone "Google like, production grade" infrastructure without the vendor lock-in of the cloud. And kubernetes on physical machines is where it shines most.

EDIT: it is very easy to complain that kubernetes is complex, but try to do virtual ips with keepalived, corosync, pacemaker to understand how much time kubernetes saves by providing the same capability in a generic way which is easy to use. I have the feeling that people take kubernetes for granted without knowing how much manual work one has to go through in order to build a similar experience using the standard building blocks available.

EDIT 2: straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google" that is a lot of saved time just by using kubernetes.

Kubernetes "builds upon" Google's "experience," but it isn't what Google runs and doesn't totally reflect what Google actually does:

https://twitter.com/copyconstruct/status/1253968037579030532

https://twitter.com/copyconstruct/status/1254273432847446016

If you happen to have the problems Kubernetes solves, it's great. But even Google doesn't have exactly those problems.

(And there is a huge difference between setting up Kubernetes and hosting Kubernetes. If you know how to sit in a chair, you can be in a plane's pilot seat just fine, but piloting is an entirely different skill.)

Correct, kubernetes is a destilation of those 15 years of experience acquired running something else that also has network, storage and compute concepts. I didn't said Google uses kubernetes, though :)

Still, kubernetes makes a wonderful job abstracting all the hard work of using the underlying tools.

Look how simple is to create a ipvs round-robin by just defining a "kind: Service" object, try to do the same using the bare tools in a way that is synchronized across N computers.

I would argue it doesn’t go far enough, especially in networking/discovery. Kubernetes is incredibly useful but I personally want more out of the box than just abstractions over existing linux technologies.

Networking just works.

Discovery via DNS is pretty standard too.

Mind to elaborate on what you would like to see in those areas?

I don’t know enough about kubernetes networking to really give it a thought. That’s my issue, it’s kinda a really grey area. I just don’t know enough about it to be able to judge what I’m missing. Does that make sense? That’s why I want to see more.

Discovery over DNS suffers from the TTL issue right? You’re basically dos’ing yourself with DNS updates?

I don't think "lessons learned from Borg" is necessarily a recipe for success. Omega was also supposed to be lessons learned from Borg, but it actually sucked and never launched, with only a few bits of Omega getting cherry-picked back into Borg.

Also, I would disregard the "moving toward" in that post. Borg has always had mutable container limits, for 10+ years (Large-scale cluster management at Google with Borg, §5.5).

> setting up kubernetes is as easy as "# apt install kubelet kubectl".

That's not even remotely accurate unless you run Kubernetes on AWS, Azure, GCP.

> straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google".

Sounds like you need to read that sentence carefully. They never say they use Kubernetes.

It seems to me you're a Kubernetes fanboy.

Reply on news.ycombinator.com