Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data.
[1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first.
More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
* BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues
We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.
E.g., as of right now, having to concatenate hundreds of columns manually just in order to pass them to some ml library in a contiguous format is always a pain and often doubles the max ram requirement.
```
import pyarrow as pa
my_col_of_3x3s = pa.struct([ (f'f_{x}_{y}', pa.int8()) for x in range(3) for y in range(3) ])
```
If using ndarrays, I think our helpers are another ~4 lines each. Interop with C is even easier, just cast. You can now pass this data through any Arrow-compatible compute stack / DB and not lose the value types. We do this for streaming into webgl's packed formats, for example.
What you don't get is a hint to the downstream systems that it is multidimensional. Tableau would just let you do individual bar charts, not say a heatmap, assuming they support rank 2's. To convert, you'd need to do that zero-copy cast to whatever they do support. I agree a targetable standard would avoid the need for that manual conversion, and increases the likelihood they use the same data rep.
Native support would also avoid some header bloat from using structs. However, we find that's fine in practice, it's metadata. E.g., our streaming code reads the schema at the beginning and then passes it along, so actual payloads are pure data, and skip resending metadata.
* BigQuery: no
* Redshift: no
* Spark SQL: no
* Snowflake: no
* Clickhouse: no
* Dremio: no
* Impala: no
* Presto: no
Disclaimer: I am a member of the TileDB team.
They are stored in tables in efficient column-oriented format.
There's an interesting query language, SciQL[1], built to support such use cases. It can be used with MonetDB.[2]