Hacker News Reader

Comment by fulafel | Hacker News Reader

Desktop PCs have about 35 GB/s of memory bandwidth and can do compute at ~200 Gflops, so this is just ~10% of peak bw and leaves you a budget of 200 flops computation per float value. If all 4 columns are accessed, there is still enough bandwidth (no idea of the data here was columnar layout or not).

The relevance to big data or out-of-core computation is left hazy, which would make this I/O bound in most cases? 4 GB fits easily in memory and is just mmap'ed from the OS disk cache if the data was recently touched. I guess with 4 columns you get to 16 GB which might be pushing it on a laptop.

View on HN

You are right, I'm actually underselling it. 1 second is the typical performance for doing a 2d histogram (or other binned statistics) since it involves writing to memory as well.

I just ran a quick benchmark: In [7]: %timeit -r3 -n3 df.mean(df.ra) 330 ms +- 5.46 ms per loop (mean +- std. dev. of 3 runs, 3 loops each) In [11]: f'{len(df):,}' Out[11]: '1,692,919,135' In [12]: 330/len(df)1e9 Out[12]: 194.92957057278463

so it is 0.2second for 1.7 billion rows, which is:

In [15]: (len(df)8/10243)/0.2 Out[15]: 63.066152296960354

63 GB/s. (this is a high end machine, on my laptop I get ~12GB/s)

We do not use float32 much in science since you really should know how not to screw up. It does give some extra performance boost (not much though), and also saves you on memory cache.

Is this cold data? Or already in RAM? What about a billion rows that are not in RAM yet?

How does it compare to plain numpy or pandas?

My thought was on first access or out-of-memory sizes. This would always be bound by I/O which means it is kind of a meaningless statistic.

Don't get me wrong, this seems like a project I will use but that marketing speak is weird.

Don't expect this performance on a 4GB machine. Most machines now would have 16GB, or more. Let us assume you have 32GB and take a 24GB dataset. Most libraries load this into memory (allocating 24GB, leaves max 8 GB for the OS, including disk cache). The next process that wants to do the same cannot without a memory error. Also, when you restart your program, the OS will not have it in the cache, it will be as fast as your hard drive.

Vaex is much smarter with memory, it memory maps the data, nothing is allocated, all the memory is left to the OS for disk cache. This means you can have 10 users regularly restarting/running their program, without any (or minimal) harddrive activity. So I think it is a fair statement, and actually very conservative (see my other comments). An important message is to make people aware that working with a dataset this large is a pip install away, no need to spin up a cluster (yet).

Reply on news.ycombinator.com