Hacker News Reader

Comment by liuliu | Hacker News Reader

With multi-thread and some tricks, csv parsing should be able to reach SSD speed (~2GiB/s read): https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...

At that point, I think the speed is limited by boxing JS objects.

View on HN

nice, is this something you've tried in WASM or WebGL/WebGPU?

at some point it makes sense to simply recommend using WASM SQLite (or DuckDB) to load the data for OLAP stuff. im not gonna encourage anyone to use JS on multi-GB files. even a few hundred MB becomes questionable ;)

The CSV parser for [Perspective](https://github.com/finos/perspective) uses the [Apache Arrow](https://github.com/apache/arrow) C++ CSV parser compiled to WASM. It's not currently multi-threaded but this is possible as well to my understanding.

what i learned from benchmarking Dekkai, is that simply "being in WASM" is a poor indicator of actual performance if you ever need to pull the values across that boundary into JS...especially strings.

i'll take a look and see if i can incorporate Perspective's parser into the bench, thanks!

What does it mean to box an object?

I believe allocating memory for objects on the heap. https://stackoverflow.com/a/80113/10654749

I think it’s more specific, something along the lines of turning a reference (pointer-allocated) object to a value-based object? But I’m woefully ignorant on this.

Oh, by "boxing object", I meant to create the in-memory object that can be used by JavaScript runtime, preferably native objects. For example, in JSC case, something like https://developer.apple.com/documentation/javascriptcore/jsv....

Ah, thanks

Reply on news.ycombinator.com