Hacker News Reader

I think the biggest issue is provenance and access control/logging—Drive doesn’t have a lot of great tools for that. Backups as well.

That said, I’m struggling to imagine how you could even efficiently process on the order of 1 PiB of data from Drive. Having that in a data warehouse or some object storage close to your compute seems required to make any good use of it.

View on HN

Logging in Drive is pretty robust (backups are not). There's a few tools that can move that level of data, but google's now-first-party migrate product could likely handle it.

That logging will give you who/when but won’t have things like field level access logging or information about the analysis pipeline itself.

With regards to data locality, I was more thinking about running a map reduce or similar ETL pipeline.

Reply on news.ycombinator.com