I'm tempted to benchmark this against better known tech, maybe anyone has some insight to share?
And a 4 node Spark cluster with Parquet or Arrow files and a Scala job will compare favorably to this because it's also a lazy evaluator and the benchmark problems here are embarrassingly parallel.
PS this is a very cool project!
I think they solve a different problem, smaller data, data/relational integrity, data mutations. Dataframes can take shortcuts here.
If you want to do a serious benchmark, feel free to contact us. Github: https://github.com/vaexio/vaex/issues Email: (I'm easy to google).
It would indeed be interesting to see how this approach with Vaex compares to Postgres. Though, I would be quite sad giving up SQL in favour of Pandas DataFrame indexing and Python looping :)
We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html
I think GraphQL is easier in combinations with front end development, and you can tab-complete your way out. Early days for this sub-project, but I think very promising.
This all seems pretty interesting. I will give it a go.
It feels like growing wings and a jetpack. Almost everthing is waaay easier and faster.