HNHacker News
TopNewBestAskShowJobs

smarterclayton

533 karma · joined March 30, 2015

submissionscomments
smarterclayton··on Kubernetes: The Documentary
One of the stories I provided for the documentary that didn’t make it (but is one of my favorites) relates to an early 2015 discussion I had with Brian Grant (googler / architect / worked on Omega / also in the documentary).

Brian writes very succinct and clear descriptions of problems encountered inside google, breaks them down to “what worked, what didn’t work”, and then would summarize a design seed in a github comment (of which several such subsystems subsequently were implemented that have been fairly successful).

I would keep mental track of some of these comments and assign a score based on how much R&D investment at google was compressed into a 200-300 word comment. I guesstimated some at being valued at upwards of $100k a word, and jokingly asked brian if he was authorized to share $10M worth of google’s secret sauce. He laughed, but reminded me of some of the same things mentioned in the documentary (which we all knew in the project at the time) - paraphrased:

“if you can seed a community project that includes some of these patterns built on that research, and that enables $10 billion of application team value, and google can capture $1B of that in google cloud, that’s a pretty good return on investment.”

And definitely among the key contributors I worked with from google was the very human and very relatable (for HN) desire to build that shows those internal ideas off and “do a successor to borg that really takes what we’ve learned and doesn’t just copy it”.

Even if it was a business strategy to help catch google cloud up… the beauty of open source is that you can benefit from it regardless, and it moves the industry forward in a meaningful way.

smarterclayton··on MicroShift
When Gateway API lands we definitely plan to streamline everyone moving to that (since it’s the superset of Routes and Ingress and supports the features we didn’t want to lose going back to ingress). That should help.

We have considered having a controller that mirrors both routes and ingress to a gateway http route (since gateway is similar to routes), but plans aren’t finalized yet.

smarterclayton··on MicroShift
For context, the reason Routes were designed to inline cert info (about 6-12mo before ingress) was that ingress controllers (which run on nodes) having the ability to read all secrets (which means the ingress controller has to be considered to be as powerful as any ServiceAccount on the cluster) was too scary. The alternative chosen was to prevent ingress controllers from seeing any secret but the ones the user decides to expose via the route.

Later on, the pattern of referencing secrets in extensions became more common, and things like the NodeAuthorizer (which allows nodes to only read the secrets associated with the pods scheduled onto them) demonstrated a possible different pattern we could have chosen to implement (although nothing that can be efficiently implemented without changing kube itself today).

Agree routes should have added a ref - that was feedback informing Gateway API, and once that hits GA and provides the best of both routes and ingress, we would probably suggest using that instead. Routes is mostly frozen for all but critical new features now though so we can ensure Gateway has everything we need to replace it while still providing the necessary forward compat.

I would treat routes as sensitive. Note that within a namespace there is minimal cross user security (not part of the kube / openshift threat model), so giving namespace read access to a specific set of routes and only infrastructure users access to all routes, OR using a wildcard cert on the routers and keeping all key material out of the user’s space. On 4.x versions you could also create multiple ingress controllers and assign them to different namespaces, preventing leak between them.

smarterclayton··on MicroShift
If you use the reflection package in Go, the dead code elimination optimization step does not run if you use reflect.Method to look up a method because you can use reflect to reach those methods and the linker preserves the possibility.

Unfortunately, almost every large client library has public methods AND references someone who calls the reflection method somewhere, so in many cases you get the full tree of dependencies.

It’s unfortunate because any one dependency can bring in reflect and trigger the behavior. It hurts large generated method libraries the most since you usually only use a small subset.

See https://go.dev/src/cmd/link/internal/ld/deadcode.go line 318

smarterclayton··on Systemd, 10 years later (2020)
Depending on the container runtime in use (containerd, crio), a separate process is managing and running those container processes (so k3s is telling that process what to do, but doesn’t “own” the containers). Also, modern container runtimes interface directly with systemd to launch transient units (think systemd services but without on disk config), so restarting a cri-o for example won’t restart those processes. That allows systemd to manage the process and for k3s/kubelet/cri-o to be upgraded / crash and recover.

Under crio to stop all containers you can (well, should but YMMV depending on any changes we’ve made recently)

systemctl stop kubepods.slice

Note that doing that a subsequent start shouldn’t bring the containers back unless k3s and crio were both running (the transient units don’t survive restart).

Note: there’s tons of complexity here that has changed over the last five or so years as container runtimes evolve, so if anything I said is wrong now it was right at one point and might actually be a bug.

Kube for instance didn’t use to gracefully shut down containers on node reboot, and over last few years we’ve gradually fixed all of the complexities so that you do get the correct shutdown behavior between systemd (which takes the “power down” input) to the unit (which encodes how long graceful shutdown should take based on the pod termination grace period seconds) to cri-o (which has to set the correct behavior in systemd) to kubelet (which has to stay running to keep updating the API) to the other components on the node (like container networking, which needs to stay up until all pod network pods are done).

systemd makes a lot of that a lot less painful than it would without.

smarterclayton··on Lego has designed a set that can't be taken apart
The Lego Star Destroyer has several “hammer of Thor” greebles (the interesting decorative surface details) in the engine section
smarterclayton··on A Convivial Chap Led Easily Astray: On G. K. Chesterton
Is there a supporting citation for the 85% number?

I looked at https://en.m.wikipedia.org/wiki/Jewish_Bolshevism and this seems to be a specific factual inaccuracy (since that number seems hard to believe on the surface based on demographics from that era) used to make what seems to be an invalid argument.

smarterclayton··on Kubelet spends 20% time in GC in most environments where it runs
Right now it’s mostly the CRI/kubelet interaction (which polls the container runtime for containers) and metrics scraping generating the garbage.

Having an incremental CRI watch (vs poll) has been on the backlog for years, but correctness and features have been more important for most people.

Several people were looking at this recently as we were squeezing management pods + system processes onto two dedicated cores for telco/edge real-time use cases (the other tens of cores being reserved for workloads). It’s not that hard to fix, it’s just never been the top priority.

Real software has curves.

smarterclayton··on NSA Kubernetes Hardening Guidance [pdf]
Most (but not all) overlay networks are implemented in kernel. If you compromise one node in a cluster, you can fairly trivially snoop traffic, bias other nodes to send traffic through you, or listen via various mechanisms such that you can intercept traffic flowing between workloads not actually located on the compromised node.

So always encrypt everything unless you’re in a very rare environment with central network control that cannot be compromised or intercepted from a given machine.

smarterclayton··on NSA Kubernetes Hardening Guidance [pdf]
And making it so that "many clusters" look exactly like "one cluster" is one of the goals the kcp prototype was exploring (although still early) because I hear this ALL the time:

1. 1 cluster was awesome

2. Many clusters means I rebuild the world

3. I wish there was a way to get the benefits of one cluster across multiples.

Which I believe is a solvable problem and partially what we've been poking at at https://github.com/kcp-dev/kcp (although it's still so early that I don't want to get hopes up).

smarterclayton··on MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
Other than bugs around bad connections causing hangs (the kubelet is less vulnerable to pathological failures in networking causing it to stall), nothing significant.

Kube is designed for nodes to have continuous connectivity to the control plane. If connectivity is disrupted and the machine restarts, none of the workloads will be restarted until connectivity is restored.

I.e. if you can have up to 10m of network disruption then at worst a restart / reboot will take 10m to restore the apps on that node.

Many other components (networking, storage, per node workers) will likely also have issues if they aren’t tested in those scenarios (i’ve seen some networking plugins hang or otherwise fail).

That said, there are lots of people successfully running clusters like this as long as worst case network disruption is bounded, and it’s a solvable problem for many of them.

I doubt we’ll see a significant investment in local resilience in Kubelet from the core project (because it’s a lot of work), but I do think eventually it might get addressed in the community (lots of OpenShift customers have asked for that behavior). The easier way today is run edge single node clusters, but then you have to invent new distribution and rollout models on top (ie gitops / custom controllers) instead of being able to reuse daemonsets.

We are experimenting in various ecosystem projects with patterns would let you map a daemonset on one cluster to smaller / distributed daemonsets on other clusters (which gets you local resilience).

smarterclayton··on Everything useful I know about kubectl
Wow, I’d forgotten about this. The reason no one has fixed it is partially because I didn’t do a great job of describing what the fix was I expected to see (clarified now). Reopened and will poke folks to look.
smarterclayton··on Secret Management in Kubernetes
I’m pretty sure the author just wasn’t as familiar with the topic as they should have been. Should have been caught during editing though.

As Joe notes, this was just how you most easily encode binary data in JSON. We did discuss that the encoding has a minor benefit to “over the shoulder reader” obscurity, but that wasn’t the goal.

smarterclayton··on You use more open source software than you think
It’s not a requirement at Red Hat, because anyone can be converted to the open source religion. This surprised me when I started - I thought I would need to be indoctrinated, but then it just became normal to see open source as the only way to work.

That said, RH hires key contributors all the time (as do many companies), so being the maintainer of a project is definitely a plus.

smarterclayton··on Apache Mesos to be moved to Attic
I worked with tech companies and some banks in the early days of Kubernetes who were deep in Mesos and many of them felt the same way. It’s really hard to see this as it’s happening.

Kubernetes is absolutely “worse is better” when it comes to scheduling. But scheduling isn’t the problem most people needed solved. Kubernetes standardized deployment patterns of multiple apps well for most people. Everything else was a nice to have (scheduling, scale, ease of initial setup, etc).

smarterclayton··on Apache Mesos to be moved to Attic
As an early participant, while someone might have had this plan, I certainly did not hear this ever stated by anyone. Having the ability to run on Mesos was discussed as interesting and in the spirit of community but the work we did treated Mesos as a source of learning but not as a serious thing to integrate. I’m sure others considered it, but as justicezyx says it was an inclusive idea but at best something a few people cared about.
smarterclayton··on Ask HN: Which are the best Go repositories to read to learn the language?
That’s not true, because I’m the original author of ObjectMeta and I did it in golang. :)

Half of the point of Kube is that every object looks similar enough that you can deal with them generically. ObjectMeta is what enforces that, so yes it’s OO, if there was only one base class possible and no inheritance. The ObjectMeta as an interface construct exists for performance reasons in Kube vs the reflection we used early on (which was mostly due to people learning Go and dealing with limitations of the language as we rapidly prototyped new ways of building out our object patterns).

A prototype of kube was done at google in Java, none of that survived translation to the open source project (edit: or was rewritten prior to v1 as we matured the subsystems that replaced the basic concepts in the day 1 project, like clients writing directly to etcd). Many of the early authors had lots of Java experience, but there was never any attempt to bring Javaisms into Kube or any Java to be converted, just conventions.

EDIT: actually, metadata/spec/status is more a repudiation of Rails and the active record style “hundreds of properties with no organization” JSON apis. We wanted to clearly separate metadata (common), spec (desired intent), and status (actual state). That led naturally in go to an embedded struct for ObjectMeta and then the interface for getting access to the generic properties on any object.

smarterclayton··on D2iQ (formerly Mesosphere) sunsetting DC/OS and focusing on Kubernetes
We (Red Hat) sat down with the Mesosphere team pre-Kubernetes and were seriously considering Mesos and Mesosphere to replatform OpenShift (our PaaS) onto. When Google decided to make a run at Kubernetes, we decided to switch, despite having quite a few larger customers (financial, animation studios) urging us to go down the Mesos path. For us it was less history and accumulated baggage, the opportunity to learn from previous systems, to make the container spec a bit more core to the platform (since Mesos already had fairly opinionated choices in place), and to be less “high scale scheduling” and more “application infrastructure for the masses”.

I respected the choices and commitment the Mesosphere team had made, and their execution always reminds me that sometimes you have to commit, sometimes you have to pivot, and sometimes you don’t know which way is going to pay off. We might have gone down this path as well, and sometimes you’re not in the right place despite everything.

smarterclayton··on Canonical introduces high-availability Micro-Kubernetes
Yes, separate legal entity, combined financial entity (although I know little about the financial details).
smarterclayton··on Canonical introduces high-availability Micro-Kubernetes
We talked about it recently, and certainly people want to do this, but I don’t see it happening anytime soon in core. Downstreams are free to carry patches though and many do.

It’s more that the core team doesn’t have the time to support these, and we already have a fairly tight contract with storage, and we still find edge cases that need to be fixed, so the extra cost for the committers to support this is high.

smarterclayton··on Canonical introduces high-availability Micro-Kubernetes
If you run etcd as a singleton you already get this behavior - consensus is shortcircuited and you only have to pay the cost to write to durable storage (which is already heavily batched). And you need durable writes so you can crash recover (kube has a gaping hole today in that a restore from an earlier time point breaks many controllers until a reconcile is performed).

Note that Kubernetes requires a total ordering of writes (which simplifies how hard it is for us puny humans to reason about) AND requires strong consistency in order to provide guarantees like “this pod only runs on one machine at a time” and “PVs aren’t released until the pods are really stopped”. Leaderless is a simple tradeoff - singleton or three instances. That’s the best possible choice in the world and it’s etcd and it’s relative simplicity that make it possible.

I’ve never seen a production Kube system with HA etcd go down due to non-human error, so I don’t believe single instance is going to give you better availability when single machine faults happen (they happen frequently; about 0.5-1% of the machines in the OpenShift fleet - cloud and on-premise - are down at any one time due to power, software, or hardware issues). Almost all of those clusters tick along fine when they lose that machine.

smarterclayton··on Canonical introduces high-availability Micro-Kubernetes
Wow, I’d didn’t realize the 4.1 docs didn’t get updated - newer doc versions are correct in that disconnected subscriptions just manually entered via OCM. 4.1 didn’t include the support for disconnected clusters and at the time the docs were correct. Newer versions are more clear.

https://docs.openshift.com/container-platform/4.5/support/re...

I will follow up on opt-out being the default for the evaluation version of OpenShift. It is already opt-in for OKD.

smarterclayton··on Canonical introduces high-availability Micro-Kubernetes
As the person who has worked on making sure telemetry is useful and valuable for users and customers (helping drive insight into quality of kube and close the loop on fixing persistent issues), can you provide some more details about how we’ve let you down? We tried to be as responsible as possible but obviously we’ve failed along the way - what can be done to improve it?
smarterclayton··on IBM looking for 12 years’ experience in Kubernetes administration
All of the patterns and default code referenced there were things I wrote or helped write or reviewed, and while a certain “java patterns” mindset was common in early devs, I can assure you we didn’t transcribe it from Java, we simply used patterns we were familiar with from previous languages. None of the open-source Kube project was “transcribed from Java” to the level being described here.

Kris isn’t wrong, but the details are far more nuanced than that. This talk is a non-primary source :)

smarterclayton··on SUSE to Acquire Rancher Labs
Yes, to run tests that root your whole cluster, the test running for conformance grants “root your cluster” permissions.

I occasionally regret the defaults we picked because people get frustrated that random software off the internet doesn’t run.

That said, every severe (or almost every) container runtime vulnerability in the last five years has not applied to a default pod running on OpenShift, so there’s at least some comfort there.

To grant “run as uid 0” is a one line RBAC as assignment. To grant “run as uid 0 and access host” is a similar statement.

https://github.com/openshift/origin/blob/master/test/extende...

smarterclayton··on Building a high performance JSON parser
A concrete example here is that almost everything in the Kubernetes ecosystem (the control plane, operators, integrations, etc) is CPU bound by time spent Marshalling/Unmarshalling JSON or protobuf. So in some domains it’s all that moves the needle (in Kube I personally spent several years in protobuf, alternate JSON impls, fine tuning the hot path). Some problems are marshalling bound, although it’s not all of them.
smarterclayton··on Spark joy by running fewer tests
Of course, to be able to see the signal, you have to be ruthless to noise. Deflake first, disable if you must, but never stop until it goes green and then be ruthless about keeping it that way.
smarterclayton··on Spark joy by running fewer tests
There’s one particular domain where this question runs deep - tests that verify parts of distributed systems. Note: please do not extrapolate any problem described below outside of distributed systems.

For instance in Kubernetes we run lots of tests that verify parts of the system actually working together, not just mocked. A fair amount of the time, once tests are in and soaked, the test flake is actually the canary that says things like “no, the kernel regressed networking AGAIN” or “no, your retry failure means that etcd lost quorum - which shouldn’t have happened, what on earth is going on, oh, someone suddenly started hotlooping and now the api server is performing 10k writes a second”.

At some point, testing the system in isolation is not enough because all failures are interactions of multiple components. Your flakes are the evidence that the gigantic stack of software known as “modern computing run from HEAD lol” is full bugs.

It’s definitely a culture change - once anyone anywhere thinks “oh it’s just flaky” they stop treating it like signal. Once they treat it like noise, it’s very hard to unwind.

Today, when I look at one of those test suites and see a flake, I assume that everything in open source is broken, again, and pull out the shovel, because 75% of the time it is.

smarterclayton··on Featherweight Go
Kubernetes had a non-public prototype in Java. None of the code in the main tree even resembles the code from that prototype (I was one of the first non-googlers to contribute).

It’s just ugly because that’s how it evolved after a community formed and that’s what real evolved software looks like. :)

smarterclayton··on Featherweight Go
Kubernetes has lots of workarounds - I’m not sure generics are the ones I’ve worried about the most.

If I had to write a list of “tools I would have preferred to have”:

1. Perfect serialization libraries - Serde is the gold standard here, and IIRC the macro system from Rust is as important a part of it as the traits part.

2. Mutability of types in shared caches - to build efficient controller patterns you need fast and efficient caches - an immutable type that protects against mutation would have saved a lot of ugly mutation testing code

3. Client library definitions shared across types - we could have reduced a fair amount of boilerplate with generics, although most bugs are in the serialization part these days as types evolve

But most of these are dwarfed by the need to scale reviewer time across hundreds of people submitting PRs that change fundamentals of the code. The simplicity of Go and ability to review stupid, obvious, absolutely critical code is the reason why I love it.

Having a strong way to reduce cleverness has been a godsend. It doesn’t mean cleverness is bad, or I don’t want more tools in Go. But I appreciate that aspect of Go in a way that others may not.

Edit: I have been watching kube-rs mature, but to get the same level of scale and performance we’d have to reproduce the shared index informers and that’s a lot of drudge work to get right. That’s one place where someone motivated could really help Rust become a first place way to extend Kubernetes, which would be exciting.

← PreviousPage 2 of 7Next →