Yes this has been an up-and-coming theme in the data science world. Arrow for the data format, Ibis for the API, Acero/Velox/DataFusion/DuckDB/Polars for execution, Substrait for the query plan representation, etc.
Isn't spark already providing a bundled and somewhat functional version of all of this ?
Right, so I think the idea is to "unbundle" this so that you can compose your own data analytics engine.
By the way, we are looking for talents of velox, data fusion, ducdb, clickhouse to build the world's fastest vector db milvus(mainly on search side). Contact me if you have this background!