Implementing join planning in our open source Golang SQL query engine
dolthub.com
dolthub.com
With pitfalls explained and solution offered.
Therefore, I believe all query engines should allow the changing of a plan mid query. Ie., when a query or subquery is not producing matching rows as rapidly as predicted, the query should be paused, a new plan selected, and the query continued with the new plan.
My testing of this on a fork of postgres found some queries can see a 1000x speedup, especially ones with tens of joins.
This leads to some complexity around cursors, but in my view, this complexity is worth it.
that is much more complex while in principle similar to the well known more practical approach - run a quick sampling of the related data and choose the plan according to the sampling results.
I want exactly that but a) for postgres and b) written in javascript/node so I can use it for in-process access while running test suites.
I know they speak mysql as a wire protocol, just would be cool to avoid another process/port to manage.
Basically, it's a custom version of PostgreSQL that supports git + Docker-like operations (versioning, sharing, branching, delta compression, Dockerfiles for datasets etc). It's not in-memory but it runs in Docker and has some tooling built around it to make it simple to spin up and manage a throwaway DB for integration test purposes.
Here's an example of it in action in use cases like machine learning [1] where we can switch between different versions (test/train) of the dataset without having to change the client application.
[0] https://github.com/splitgraph/splitgraph
[1] https://github.com/splitgraph/splitgraph/blob/master/example...
By the way, I can't click your about page, or the menu button (Firefox android). It just refreshes the page and goes to the data repositories on your main page.
https://www.dolthub.com/blog/2020-03-30-dolt-use-cases/
I've forwarded your bug report to our web team.