cfors

Seriously. Especially when something seems like a bizarre choice in the code, I'll provide a link if there is a corresponding GitHub issue or code regarding it.

I've never had an issue with GitHub issues regarding linkrot, but it happens quite frequently when linking actual source code, especially code not in your control. Not sure what a better solution is for that, unfortunately.

I generally justify it by the anecdotal observation that the code is generally not being used anymore by the time the link dies out, but that is far from an ideal justification.

Imagine working at a company, that has these gigantic valuations, and you hold stock in this company. Since you probably don't have the insights to the actual finances of the company you, like everyone else, base the company's valuation on the private market. Then only to find out that your shares you thought were worth $100, are now worth $15 (except much worse).

In the meantime, your CEO walks away with over a billion dollars in his pocket on a handshake deal to basically shut up and leave.

I would be furious if I was an employee that dedicated time and energy to this company.

An interesting side-effect of having awesome tooling. When you have a service that is taking too much memory and can see directly on a Grafana dashboard that the reason your service keeps getting OOM'd because of too many Goroutines you can avoid even having to go into a box to debug.

When I used to be on call, I remember very distinctly an incident where there was a ton of restarts on containers for a set of nodes (This was on K8s, so not immediately obvious). The operations team mentioned to me that in the logs (K8s logs here) there was a lot of errors being generated along the lines of "Discovery failing, restarting".

I asked if we had a tcpdump if we could see what was happening, and the answer I got was "There is no tcpdump in our container." Meaning that we didn't have the binary in the actual container image.

For anyone with any knowledge of Kubernetes, you know that you can easily just SSH to the machine running the pod, get the current port it's being ran on and then run a tcpdump on there. However, the fact that all of this couldn't be tied together to come up with a tcpdump to understand that caused this issue to persist for an extra 3 hours when the issue was a flapping NIC that was misconfigured.

Without understanding how your systems actually work under the hood, you will be running them with your hands tied. It's not good enough to understand one abstraction, you need to be prepared at anytime to peel back the layers of abstraction and get your hands dirty in a different layer.

Good question. I have had all-hands in the past that were used to hear the intended direction of how goals were going to be measured, which was useful since there is always some whisper-down-the-lane noise for the actual message. But I have also been a part of all-hands that were just PR fluff about what a great job everyone is doing. So, I guess as most things, it depends.

I'm not saying there isn't a use case for this, it's just that for the most part I prefer dumping what I have from an immutable git commit into the Kube API. If something needs to happen after the fact for governance or injecting sidecars, so be it but that is out of the scope of what I am deploying.

I am aware of that, and here is one the of the issues I have with the strategic patch strategy:

> The patch you did in the preceding exercise is called a strategic merge patch. Notice that the patch did not replace the containers list. Instead it added a new Container to the list. In other words, the list in the patch was merged with the existing list. This is not always what happens when you use a strategic merge patch on a list. In some cases, the list is replaced, not merged. [0]

This seems like a lot of cognitive overload. I understand there are some use cases for this but really, all I want is to have my K8s resources all tracked with a git commit reference and then that is what is deployed exactly.

[0] https://kubernetes.io/docs/tasks/run-application/update-api-...

I suppose if you're not using git-ops related deployments, but Git has been proven to work well for this exact process. If you are tracking your releases as part of a Git commit, then you rollback with that.

Doesn't seem that complicated, but maybe I'm missing some benefits of this strategy that others with more experience managing K8s resources have had.

Edit: Retroactively speaking you also don't need to leverage etcd to store n*x your K8s resources in a secrets file where n is the potential rollback number, it's all stored in a Git repository.

It sure is a leaky abstraction [0]. I would argue that the benefit of it is when spinning up your first Kubernetes cluster, the many community charts created by default can be installed easily with a > helm install stable/mysql.

However, it fails when the reality of running a service in production hits and you need to dive into changing a Chart template, requiring knowledge of not just the Kube API but also more often than not ugly template-fu to get the changes that you want reflected.

I really just recommend using raw K8s resource files for almost everything unless you are managing hundreds of similar releases.

https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

This is a big step forward for Helm. Tiller has been a major pain point in the past, although I understand the design decisions why it was placed here in the past.

> In Helm 3, we now use a three-way strategic merge patch. Helm considers the old manifest, its live state, and the new manifest when generating a patch.

I will have to test this out, but honestly I think that any sort of "strategic" merge is probably the wrong approach to take when dealing with so called "idempotent" infrastructure.

Slightly off topic but I love Adrian Coyler’s blog. Since I never pursued graduate studies in CS I never really got into reading research papers but would love to start reading some on my commute to work.

Does anyone have any recommendations for finding interesting papers? Do I need to buy subscriptions? Is there a list of “recommended” papers to read like we have with programming literature e.g. The Pragmatic Programmer?

At any sort of scale (which I have to imagine netlify is running at), this will be a software issue since hosts are designed to fail.

Serving static content is crazy easy for computers, my guess is a bad configuration got pushed out.

The same issue occurs with Node and NPM. I don't want to speak for Ruby as I haven't used Ruby extensively but I can't imagine that Gem's and Ruby versions don't have the same issue.

Honestly virtualenv's solve most of these problems. If using a virtualenv seem to low level or a pain, I can't recommend Poetry [0] enough. It uses virtualenv's in the background, and has from my experience a great dependency resolver.

[0] https://poetry.eustace.io/

As another resource, I just finished up the chapters of The Linux Application Programming Interface [0] that described this.

Highly recommend this book, its filled with fun tidbits of Linux-lore.

[0] https://www.amazon.com/Linux-Programming-Interface-System-Ha...

This was my takeaway too. Too much talking about doing at big corporate companies, and not enough actual doing. Drives me up a wall.

It’s a business play. If companies don’t want to move off 2.x and are willing to pay a Software vendor to backport security fixes so that they can CYA, so be it.

I would never recommend it, but the thought has crossed my mind (definitely more than once) that the scream test could work wonders with cloud infrastructure [0].

[0] https://blogs.msdn.microsoft.com/anitag/2012/06/07/i-scream-...

edit: I suppose Chaos Monkey could be purposed for this task in hindsight.

> If you're using things which do macro expansion or anything else that involves you writing format A and it generating format B which actually does the work (or worse yet, gets turned into format C), you really owe it to yourself to run a 'diff' on the before-and-after versions of the output from the tool BEFORE it goes and takes any action on your behalf.

Seriously. This is a huge issue with Helm templating Kubernetes resources definitions. When you make a PR on your infrastructure repository, there could be many changes underneath the hood in a Helm chart that Are invisible from changes to a values file.

We had a rule in my last job the diff of the actual resource files had to be included in the PR in order to be approved because we were bit by the same.

Naming something popular != market share.

I think you have it reversed, AWS accounts for most of Amazon's profit and is used to fund other ventures.

> AWS revenue came accounted for 13% of Amazon’s total revenue. Of Amazon’s total $3.1 billion in operating income, 52% came from AWS. [0]

[0] https://www.cnbc.com/2019/07/25/aws-earnings-q2-2019.html

Pretty much all of this. If I could have sandboxed dev environments, a terminal interface is all I really need since I do most of my dev work in a terminal. I suppose some nice-to-haves would be the ability to spin up local servers that I could access from Safari.

Cool, but nothing about any focus on making the machine realistic for development.

My current MacBook Pro is getting a little long in the tooth and after suffering through the 2016 and later versions at work, I will not be buying one of those until the keyboards are fixed. If iPadOS has first class terminal support I would consider making it my full time machine.

Looks like the waiting game will continue.

Pretty cool stuff. I'm assuming you have to set up Antenna's where you have your servers colocated and monitoring the transport stream's going through?

Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.

Couldn't agree more, but honestly I think that is a completely overplayed issue. Just be explicit (This requires Python3.7) and have a requirements.txt file, maybe bundle with a `make install` command and move on.

If people can't figure that out... I don't know how you're going to expect them to read a cryptic bash script.

Don't get me wrong there is totally some good use cases for bash. Init-scripts come to mind when you don't want to lug around a Python VM in a lightweight container, for example.

My disclaimer is that I love bash. I write way too many things in bash. But eventually I go back and refactor the scripts into Python (or lately, Go) because I am so sick of people refusing to touch my bash scripts.

String munging in Python is so much objectively easier to read than bash. I would say that it's because it is similar to a lot of the more popular languages than the sort of cryptic parameter substitution bash provides. And if that makes it easier to maintain some automation, then its worth the effort just to use Python IMO.

Both of your examples made sense to me, but when you take a step back that sentence doesn't really mean anything without context.

A "resource" === "endpoint" in the sense that the endpoint defines where you can define the configuration of a specific resource. They are being specifically vague I am assuming because of the concept of CRDs.

But without that sort of knowledge ("Hey, Kubernetes has a customizable API using custom resources!"), that sentence is rather hard to unpack.

edit: Thinking about it some more I suppose an endpoint is an overloaded term w.r.t. Pod IPs.

Unfortunately Slack is where documentation goes to die. When I face a weird internal issue the first thing I do is search:

> in:@<channel> <error>

You would be amazed at how much unofficially documented stuff is sitting in Slack. It is unfortunate but paying attention to the right Slack channels at work can be very important in understanding company strategies and decision making, as well as random tidbits on architecture decisions.

Of course I would prefer official documentation but sometimes you gotta make do.

The control plane is an API server that distributes configurations to lightweight proxies (Envoy, in this case), also known as the data plane.

If you think about it Kubernetes concepts, the control plane is the API server and the data plane are all of the "kubelet", or worker, nodes

This is the way forward for service meshes. Consul connect is doing something very similar.

The early service meshes were all built on top of Kubernetes, due to the fact that k8s/etcd is an easy platform to build control planes on top of (SD, networking, etc. already done) but very few large companies are ready to move everything over to Kubernetes.

However, the one thing that every service mesh promises is that the clients don't need to do anything. Except that they do. End users need to rip out functionality that is coupled already tightly to the application, such as authentication, "smart" routing and metrics. They need to orchestrate tracing to pass on the headers (how do you know what routes in your application were bad if all you have is a black box!). It needs to play nicely with already built in enterprise wide things.

I'm not sure that I'm sold on the universal service mesh unless you have a high level of engineering talent. The benefits are immense but don't expect to see universal service meshes* running at enterprisey companies any time soon.

* Universal means the entire company is governed under a service mesh, not the POC running on 3 servers for a random team :)

This is the way forward for service meshes. Consul connect is doing something very similar.

The early service meshes were all built on top of Kubernetes, due to the fact that k8s/etcd is an easy platform to build control planes on top of (SD, networking, etc. already done) but very few large companies are ready to move everything over to Kubernetes.

Very interesting, I can think of a lot of places where this would be useful for managing infrastructure.

I highly recommend reading about some of the internals, it is making me rethink a lot of how configuration should be done.

https://cuelang.org/docs/concepts/logic/