I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-...

I have a question about whether it would fit this use-case:

* I need a SUPER fast KV-store.

* I'm on a single machine.

* Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\

* Values are 4-byte ints. Maybe 3 4-byte ints.

* I have maybe 10B - 100B rows.

* I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk.

Would arrow be useful for this? Currently just using sqlite.