Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.
Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.
I'd be interested to know any benefits of converting to Parquet if you know your reader supports Arrow.
Compression. For our app, we do multilevel caches, where warm is arrow (in-memory, inflight, streaming, ...) and cold (persistent/scaleout) is parquet (ssd, s3, ...).