We build a next-gen version of HDFS (HopsFS) that has distributed, consistent, transactional metadata where small files (<1MB) can be stored in NVMe disks in the metadata layer. And it's open-source - cf WekaIO, etc. And it's performance is backed by peer-reviewed publications at tier-1 conferences (Usenix fast, ACM middleware, etc).

https://www.logicalclocks.com/millions-and-millions-of-files...

Our business model is to build a data science platform, Hopsworks, around our distributed metadata layer. And yes we use YARN (training models) and also Kubernetes (serving models). The choice of resource mgr is really just an implementation detail, as the platform is backed by a REST API.