ssinchenko

User friendly tooling (cargo, IDE integration), user friendly error messages.

> All of this would have easily fit in memory on any reasonable modern system.

Understand me correctly. This is my research project and I only have a laptop, not a server with 256 GB of RAM. I tested my project on a 2B graph with a hard cap of 8–10 GB of RAM. Of course it fits in memory on any modern system with 64–128 GB of RAM. The whole idea was to conduct a stress test and check how my tool works in out-of-core mode, not to prove to anyone that 2 billion edges (30 GB CSV) constitutes "big data".

Yes, it is both out-of-core and multi-processor. I created this toy project to process graphs that cannot fit into memory (in CSR format) using all available cores.

The multi-processing relies on DataFusion's Tokyo workers. The out-of-core aspect is achieved through a combination of DataFusion FairSpillPool, Sort-Merge-Join, and manually offloading everything to temporary Parquet files on disk.

Thank you!

I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I implemented my own "co-partitioning" to speed up joins and achieved a performance improvement of two times! It was also easy to modify the physical plan and declare partitioning.

Agreed, but I didn't see anything like this in popular projects such as Networkit or IGraph. If you have an example of an implementation, I would appreciate it!

The previous issues were in my mind, not in DataFusion.

I tried using DataFusion as an in-memory tool, which was a mistake. If the graph fits in memory, Networkit, IGraph, etc. will almost always be faster. These tools cannot process anything bigger than the available memory.

So, I changed my approach. I wrote my own naive "disk checkpointer," offloading everything to disk and avoiding materialization. Although I was afraid that writing to and reading from the disk would be slow, it is surprisingly fast with DataFusion. The results are impressive: fast and out-of-core.

Sorry, this post is short and not very detailed. I did not expect it to be at the top of HN and receive so much attention.

Yes, my toy tool is similar by the concept to graphchi. But I did not write the vertex-centric processing from scratch and I'm relying on DataFusion built-ins (select, join, group by, aggregate)