HNHacker News
TopNewBestAskShowJobs

v2thegreat

13 karma · joined June 20, 2024

submissionscomments
v2thegreat··on Lessons Learned from Scaling to Multi-Terabyte Datasets
Hi! I'm the author.

You're right! The context that I was talking about was actually in the GIS space! Depending on what you're trying to do, it's not uncommon to have a single dataset that's 50TB large (think: 100-500m resolution global raster with daily data for 30 years)

And that's not even considering any memory overhead when performing operations on top of that data.

That type of stuff wouldn't fit into Postgres (we tried once, a long time ago), and usually, you might be trying to use multiple datasets simultaneously, so that data adds up pretty quickly!

It's my first post (and writing) ever, so I have room for improvement. Thanks for taking the time to read!

v2thegreat··on Lessons Learned from Scaling to Multi-Terabyte Datasets
Hi! I'm the author. I've seen this comment several times and thought I'd address it here.

I should've clarified that when I say multi-terabyte, I meant 100s TBs of data. It's my first time writing anything, so I'm still getting the hang of things.

Damn, that would've been a more clickbait title, too.

Oh well.