Hacker News Reader

Comment by xiaodai | Hacker News Reader

Rust will not succeed. Ppl can't debug spakr already and rust, even fewer ppl cna debug.

Build a spark killer in Julia. Everyone can read the code.

I can't speak to the relative merits of Julia, but I am honestly interested in anything that seeks to produce a less memory-hungry alternative to Spark.

Rust, to me, seems like a natural enough choice. It is easy to mate it to other languages, including all the major data science ones, so it would theoretically work well as the basis for a distributed compute engine that has good support for all of them as client languages. Would the same work for Julia? IIRC, it's a bytecode compiled language, which I imagine would make it difficult to link Julia libraries from other technology stacks.

There is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega. It looks promising.

Databricks rewrote the Spark engine in C++ (called the Delta Engine, see here: https://databricks.com/blog/2020/06/24/introducing-delta-eng...).

Hopefully there will be some good alternatives that don't consume so much memory soon ;)

Hopefully they have all measures in place to prevent data to be tainted by C and C++ "features".

There is also https://github.com/ballista-compute/ballista (shameless plug for my own project)

It is the bytecode compiled language like any other that uses LLVM as their backend.

The alternative to Spark will be Spark, as JVM slowly improves their memory APIs from release to release.

People tend to hand wave how much it costs to reboot an ecosystem.

Reply on news.ycombinator.com