I'd imagine CockroachDB is doing something similar for distributed join.
I'd imagine CockroachDB is doing something similar for distributed join.
[1]: https://en.wikipedia.org/wiki/Hash_join#Grace_hash_join
[2]: https://github.com/cockroachdb/cockroach/pull/12221
[3]: https://github.com/cockroachdb/cockroach/pull/12221#issuecom...
Another interesting idea I read about (I can't find it anywhere online) was called "join zippering". Basically you first request the cluster to solve a join by querying and streaming the key columns from a join predicate back into the cluster itself to identify which nodes have matches and then streaming the results from each node in parallel, and doing the join in the stream.
This is hard stuff but so cool too :)
I agree! we have some semblances of pushdown filtering across aggregations and some other interesting techniques as documented in the RFC[1] that first proposed the distributed execution model.
[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/RF...