Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well.
Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings.
At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive.
If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.).
I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).