Wouldn't we need more due to more overhead due to less economies of scale?
Very likely. Public transport vs. a car type of thing I suspect.
Yeah, I think the best example is S3. Only when you have to hat much data with a Pareto access distribution can you do things like oversubscribe I/O a ton, or do clever encoding / compression schemes that wouldn’t make sense at the scales of an individual organization.
There’s also trickle down economics at work here. The rest of the industry got brotli and zstd for free. Every dwarf benefits.
It really depends on what the capacity is being used for. From my experience, everything user facing (including all web serving, storage etc.) comprises a very tiny fraction of overall costs. The real $$$ (both input and output) is from big data/analytics/ML pipelines. Start a company that doesn't do any of that, and you'll have an infinitely better energy footprint.
Would the tech dwarfs choose to store all the data they ever came across, forever?
They'd be stupid not to
That's not true. Storing data has costs. And not just hard drives. There are organizational risks in keeping data forever. Analytics could be needlessly complex with excessive data retention. Just store the data you need.
Even if they didn't, I'm willing to bet that most datacenters are dominated by compute, not storage.
This rings true. The title should really be "Tech consumption in Sweden needs half a nuclear plant". What else is new?
It’s likely we would need (much) more. If you look the efficiency of google and facebook they’re significantly better than many smaller organizations when it comes to utilization and PUE. Part of this is because they can afford to throw engineers at the problem, and a 1% savings in something like search easily makes up for an engineer’s salary. Depending on that 1% gain, it could also be applied to other BUs.