"This is not hard to do."
That's a big assumption. I've worked with systems that wouldn't have been able to provide this without a LOT of work and upgrades, and they weren't anywhere near the size if Google and Amazon.
That's a big assumption. I've worked with systems that wouldn't have been able to provide this without a LOT of work and upgrades, and they weren't anywhere near the size if Google and Amazon.
Imagine if every piece of data that involves their cloud service is accessed using a custom piece of code and spread across a multitude of locations all over their network, unsorted. Now they have to build an application which can consolidate a specific customer's logs and provide an interface for them to download it. It's certainly "doable", but it's also probably the very last feature they're thinking about implementing.
Say the logs live on disk on a filer somewhere. Need to do a security investigation? OK, dig up the file the logs are in for that particular user and grep around. Lots of disk seeks, but pretty fast.
Scaling that to support production-scale queries is a significant undertaking. (Oh, you can only do 100 seeks per second, need on average 100 seeks per query, and desire to service a low demand of 100 qps? Not gunna happen!)