mihevc

If I read the example correctly they are using long living http/2 connection to send arrow IPC. Which seems pretty pragmatic.

grpc is an RPC framework and Arrow IPC is a serialization format. You can use Arrow Flight which uses grpc to send Arrow IPC. But I suppose TypeSafe doesn't provide grpc endpoints yet?

Et tu, Knuthus?

Nice! Thanks for the context, it's great to know!

How does this compare to https://github.com/Query-farm/tributary ?

I was there a week ago and it still wasn't back. The penis room was wide open though.

Here are some cookbook examples: https://arrow.apache.org/cookbook/py/data.html#group-a-table, https://arrow.apache.org/cookbook/. Datasets would probably be a good approach for the billions size, see: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-...

I wrote my master thesis on Acer C720 (2013) running xfce. I used some octave but I mostly needed it for latex. These days I don't use it anymore because the screen is not that great but it aged incredibly well.

Yet another format is CSF (Compressed Sparse Fiber): http://glaros.dtc.umn.edu/gkhome/node/1177

It's a generalized CSR/CSC (to ndim > 2) format that uses a tree data structure where each path from root to leaf encodes a nonzero element.

Nice post! For completeness: Apache Arrow is also adding sparse tensors in c++ [1] and wrapping them with python [2] (although the documentation for Python might be a bit lacking at the moment).

[1] https://arrow.apache.org/docs/cpp/api/tensor.html?highlight=...

[2] https://github.com/apache/arrow/blob/master/python/pyarrow/t...