Before joining the department, I was part of a start-up where, with a considerably smaller team, we were able to process even larger amounts of data. We could introduce a new feature in a single day - a feat unthinkable in the department I just described. This speed was not achieved at the expense of stability but was a direct result of our choice of tools.
Addressing the aspect of "questionable design and lack of readily available support", I believe only a handful of individuals are capable of developing something on the scale of Docker or Kubernetes, let alone doing it effectively. Assuming you're fortunate enough to hire such individuals, would you genuinely want them dedicating their cognitive resources to these types of problems? Even if they are highly skilled, can their solution truly compete with industry-standard frameworks developed by a team of equally competent individuals? And even if such an individual could design a superior solution, they are now responsible for maintaining it and training juniors on a system they likely have little motivation to understand.
Moreover, the 'Not Invented Here' syndrome can lead to a cascade of more of the same. For instance, we needed to orchestrate several processes. Had we adopted Kubernetes, I would have suggested utilizing Airflow, which could have been implemented in just a few days. However, we chose to develop a custom pipeline runner, which took several months to complete.
I understand that the no-frameworks sentiment has its roots – perhaps from Java's insistence on using frameworks. However, it seems that the industry swings from one extreme to the other. Golang emerged as a reaction to a world overrun with frameworks, but it appears to have veered too far in the other direction.