andygrove

Each executor within the cluster would use Arrow in-memory design. If you have enough cores and memory on a single node then potentially you wouldn't need a cluster.

I will write up some guidance in the next few days for those looking to contribute!

This is exactly the situation I am in. We have workloads that I think we could deliver with much lower TCO using Rust/DataFusion/Ballista.

We could probably even use a single node for some of our workflows just using DataFusion directly with same performance as a distributed Spark job.

Yes contributors are welcome. I will write up some guidance in the next few days for those looking to contribute!

These are great questions and topics I plan on addressing in a future blog post. SQL is a great convenience for simple analytical queries and it can be nice to be able to mix and match SQL and other access patterns (this is one thing I like with the Apache Spark DataFrame approach).

It's true that the Rust ecosystem for data science is not really there yet and I am trying to inspire people to start changing that. I think Rust does have some good potential here.

In the nearer term though I am exploring options around support user code in distributed query execution and I don't want to limit it to Rust.

Some good points. Some incredible engineering has gone into Spark to work around the fact that it runs on the JVM.

Memory overhead of Spark particularly (not just JVM) is very high. In some cases close to 100x more memory than equivalent query execution with DataFusion [0]. Also you might be interested to see my original blog post with some of my thoughts on this [1].

[0] https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ [1] https://andygrove.io/2018/01/rust-is-for-big-data/

That's kinda what I'm trying to do here. Ballista is definitely very much inspired by Apache Spark but not a direct port. Spark has some design choices that were very much driven from being a JVM-first platform e.g. the way lambdas are serialized. When I eventually get to support custom user code in distributed execution I don't want it to be limited to Rust, but that's a topic for another blog post later this year.

I've only been using Kubernetes for a couple months so far and am still learning, but I am very impressed so far. I love the way it facilitates dev and devops collaborating and the fact that it is cloud agnostic (I can even run a Kubernetes cluster on my desktop for local testing). My opinion so far is that the distributed cluster part is really solved by Kubernetes.

Some great points there.

I have multiple goals here.

Goal #1 is to demonstrate how Rust can be effective at data science / analytics and evangelize a bit.

Goal #2 is to have fun trying to build some of this myself and improve my skills as a software engineer.

Goal #3 (getting a little more vague now) is to ensure that in some hypothetical future I can get a job working with Rust instead of JVM/Apache Spark.

I agree. Rust has a very steep learning curve compared to JVM languages. My hope in building this platform is that it can provide value to other languages (especially JVM) by taking query plans and executing them efficiently.

Benchmarks are rarely fair. They are often biased in favor of those producing the benchmarks, whether intentionally or not.

Whether you want to explore this more or not is your choice. Nobody is asking you to engage in this conversation. If you are not interested and/or are not enjoying the discourse here, feel free to move along.

Thanks eklavya but it's completely normal on HN to have trolls criticizing other people's work. Best thing is to just ignore them. They can move on to criticize other posts. Much easier than contributing.

I'm sensing that you are not familiar with Rust and the safety that it introduces compared to C/C++.

This is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work.

For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent:

https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/

"But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety"

Except for the fact that it does in some cases. Rust is definitely safer than C or C++ (unless the Rust code is explicitly using the unsafe keyword).

I admit I haven't benchmarked against COBOL blockchains.

Check out my previous benchmarks on Rust-based DataFusion vs JVM-based Apache Spark workloads. It isn't just about raw performance numbers but also about resource requirements (especially RAM requirements when using JVM or similar GC-based languages). There are order-of-magnitude differences.

This is for people who want to live in a future where we use efficient and safe system level languages for massively scalable distributed data processing. IMHO, Python and Java are not ideal language choices for these purposes.

Rust offers much lower TCO compared to current industry best practices in this area.

Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.

With further development, yes. It's just Docker containers. However, the current CLI is specific to Kubernetes.

If you weren't using Kubernetes, what orchestrator would you be interested in?

Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it.

https://www.dremio.com/announcing-gandiva-initiative-for-apa...

Although I know and work with some of the contributors from this project, I have no real world experience with Dask or Python data science tools in general.

Thanks for the link. I will read about their Kubernetes support.

My goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/

No. Dremio is a mature product backed by a company. This is a two week old PoC right now. There could be overlap and collaboration for sure via the Apache Arrow project since Dremio are well represented there and I am an Arrow committer too.

Web site is down ... too much love for Rust :-)

We're working on it ... check back soon!

These web frameworks are all about reducing learning curves and making developers productive as soon as possible. MongoDB is easy to learn and will be fine for some percentage of use cases. The developers behind these web frameworks are often not coming from an enterprise systems / traditional RDBMS background and are not necessarily aware of the devops trade-offs between MongoDB versus traditional RDBMS.

I work for a software vendor and and this is a fairly typical scenario for customers of our dbShards product, who often have an app that starts to get traction and then requires a more reliable and extensible back end database that is easier to apply schema changes to without downtime.