A billion 32 bit floating point numbers are 4 Gigabytes. How can that be processed in one second unless there was any preprocessing?
The relevance to big data or out-of-core computation is left hazy, which would make this I/O bound in most cases? 4 GB fits easily in memory and is just mmap'ed from the OS disk cache if the data was recently touched. I guess with 4 columns you get to 16 GB which might be pushing it on a laptop.
I just ran a quick benchmark: In [7]: %timeit -r3 -n3 df.mean(df.ra) 330 ms +- 5.46 ms per loop (mean +- std. dev. of 3 runs, 3 loops each) In [11]: f'{len(df):,}' Out[11]: '1,692,919,135' In [12]: 330/len(df)1e9 Out[12]: 194.92957057278463
so it is 0.2second for 1.7 billion rows, which is:
In [15]: (len(df)8/10243)/0.2 Out[15]: 63.066152296960354
63 GB/s. (this is a high end machine, on my laptop I get ~12GB/s)
We do not use float32 much in science since you really should know how not to screw up. It does give some extra performance boost (not much though), and also saves you on memory cache.
How does it compare to plain numpy or pandas?
Don't get me wrong, this seems like a project I will use but that marketing speak is weird.
Vaex is much smarter with memory, it memory maps the data, nothing is allocated, all the memory is left to the OS for disk cache. This means you can have 10 users regularly restarting/running their program, without any (or minimal) harddrive activity. So I think it is a fair statement, and actually very conservative (see my other comments). An important message is to make people aware that working with a dataset this large is a pip install away, no need to spin up a cluster (yet).