My thought was on first access or out-of-memory sizes. This would always be bound by I/O which means it is kind of a meaningless statistic.
Don't get me wrong, this seems like a project I will use but that marketing speak is weird.
Don't get me wrong, this seems like a project I will use but that marketing speak is weird.
Vaex is much smarter with memory, it memory maps the data, nothing is allocated, all the memory is left to the OS for disk cache. This means you can have 10 users regularly restarting/running their program, without any (or minimal) harddrive activity. So I think it is a fair statement, and actually very conservative (see my other comments). An important message is to make people aware that working with a dataset this large is a pip install away, no need to spin up a cluster (yet).