It's a problem with a fractal shape. It's not really specific to microservices or SV big tech. To elaborate:
1. An individual software developer might bias towards feature delivery, performance optimizations, extension hooks, tests and static analysis, or some mix of those. A more experienced developer, all things equal, is more likely to strike an appropriate balance for the problem they're working on. A wrong balance might be overengineered because it took too long for the results. A wrong balance that is underengineered, on the other hand, might fail to deliver altogether by making critical-failure assumptions that go untested until it's too late.
2. A team of developers holds all the same biases as the individuals, only now they have to collaborate. This produces a team-internal political structure that enacts some compromise depending on management's goals. The compromise may not be good engineering, and it is likely to overdo things in at least some aspect. Managers often seek solutions that tame development and allow it to proceed at some predictable rate of change, or to report a metric of progress with person-hour linear scaling properties.
3. Multiple teams of developers across a larger organization also have to engage in political compromise, and at this scale, they are beholden to build and share substantial infrastructure, because management tends to act competitively, and infrastructure projects are politically useful.
4. Across different organizations, shared formats and common data interchange is desirable, and the chiefs of each organization have to satisfy their internal political structure while simultaneously coming to an agreement with the relevant standards groups. This causes the standards to bias towards a kitchen-sink featureset.
These various factors all contribute to the "enterprise" mindset, and unless you have a visionary of the Steve Jobs type at the helm, who will happily come down from the heavens to throw lightning bolts at hapless managers and force them into compromise for the good of the company, it's more likely to appear as your org gets bigger: In the default case, nobody is incentivized to take a risk on simplification or bold, radical changes. Incremental accretion of code looks good for everyone, and allows sub-par developers to blend into the background. The middle management fights localized battles over headcount and budget, and flashy projects that covertly create large engineering dependencies give them political leverage. When a new standards proposal comes up, everyone rushes to get their fingers in it. So the systems that get built, in the end, tend to express everything while doing almost nothing. Every one of these companies has layers of crushed dreams sitting in their repository.
Because Twitter is funded at the assumed ambition level of "become Google" or "become Facebook" their organization was staffed up in anticipation of achieving that scale. That inherently creates the kind of conflicts resulting in enterprisey code. The code reflects the organization, and the organization is big, so the code is big.
At the other end of the scale, where individual developers do whatever they want as they think of it, without business interest, none of these scale concerns arise. The code can be direct and to the point. Where it has to solve larger problems, it necessarily leverages the open-source ecosystem, creating a "bottom-up" pull where the developer is more likely to submit to the restrictions of the ecosystem and in turn extend that ecosystem outwards, than to attempt a bold infrastructure project; as such, off-the-shelf options, including cloud deployments, become more and more attractive as your organization gets smaller, and for solo developers, they approach "no-brainer". But, being based on self-absorbed whims, the code is much less likely to be of value to anyone else - it doesn't answer to clients, or future maintainers, or possibly even the original developer. It's less likely to have coherent design or documentation, even if the developer aims for their best try at quality, because there isn't any real conversation around it, any probing of biases or testing against the real world. A solo coder is most likely to be guilty of coding up a fantasy, which can be good if they happen to be making a game, but is less viable for a lot of other lines of software.
So the best results tend to have a pattern of: Small, experienced teams - more eyes on the design, appropriate challenge and dialogue about process and features, but nobody getting lost in the crowd or submerged in political maneuvers. Projects with some business interest, but relatively lower pressure from the top, so that feature accretion and testing is guided by well-defined business needs, not deadline pressure, bells-and-whistles, or whimsy. Management that can define a healthy lifecycle for the software and allow it some time to mature, but also be deprecated and incrementally replaced as the surrounding software ecosystem changes.
When projects fall outside of those boundaries, bad software - of the over or under-engineered sort, or the non-valuable sort, or the under-maintained sort - tends to get made.