"So, why are we doing this?" I think there are a lot of reasons. Market share, control of technology, not invented here syndrome, ego issues, fads, shifting focus on specific set of problems. I personally think that OO proliferation (C++) and Java and enterprise mindset was the point of no return. The mental point where the majority left simplicity and maintainability out of its scope. There was (and still is) an era where complexity (and the ability to comprehend it) is a synonym of mental machismo. Simple was left as something that lesser minds/developers had to worry about whilst the 10x devs (another great proliferated myth) which everyone worth his salt should strive to be (so the myth went on), must not only conquer it but also thrive on it and create more of it. Thus you got a multitude of tooling, languages, stacks that do the same things differently, you got Godzilla OO hierarchies and architecture diagrams that look like a dozen overlapped London metro maps. Now we got microservices and k8s which in itself is a testament to irony - started as an attempt to simplify uservices/cluster mgmt and has evolved to perhaps the most complicated ecosystem ever (one has serious trouble even comprehending what most of the tooling in k8s land are supposed to be doing, much less use them and don't get me started about really understanding code in there).

I personally think more people should read and embrace the Unix philosophy of simple and reusable but on the other hand I'm afraid that the ship is unstoppable now.