Their source data, common crawl has s3 hooks, probably played some part

The main reason is that AWS S3 is widely used. We obviously want to make it work on HDFS, MinIO and other relevant storage systems.