I’ve had some bad luck with hbase for similar but not tried spark - it’s better optimized for writers than readers iirc. Pinterest[1] did an interesting thing where they worked directly on the hfiles. Everywhere I’ve worked I feel like I could pick any random data processing job that took more than a few minutes and get orders of magnitude improvement, there’s just so much inefficiency

[1]https://medium.com/pinterest-engineering/open-sourcing-terra...