MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark
dmux.github.io
dmux.github.io
I work on connectors to various datasources, mostly SQL DB's
The most useful tests are the integration tests, sending actual queries to the source
For most DB's we have a giant Docker Compose which runs a local instance
But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance
I would kill to be able to run test versions of all of these locally
Our connectors receive a generic query IR from a Datafusion-based engine
Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source
For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc
Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test
I recall seeing a project based on DuckDB parser and engine which emulated dialects for one of these but because the sortinh and other behavior is different it wasn't usable
With a list of "Known gaps" and implemented engines listed with status like "Real", and of course the Quick start first, as Claude loves to do.
It's not "wrong", it just shows low effort. But then again it's just something someone needed and shared it for free so let's not complain too much.
All the products and saas launching with obvious AI landing pages, that the authors did not even care to trim down, that's another story.
Not that I'd use it, I want to avoid Databricks - the Duck ecosystem is far more compelling for my modest needs, and handles local develop a lot better (IMO).
I wonder if there is any plans on extending it further? (just a bit)
The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames.
Also looks like they did one full round of english alphabet and are on series L second time around. I feel like we might have some a tech theranos on our hands here.
Spark is such a headache to run that people would rather pay Databricks to do it for them. They also have some secret sauce like Photon (C++ version of the Spark engine) and the new RT Lakehouse warehouses (10x or so speed up on read-only work flows in my experience) as well as good proprietary optimization features like Liquid Clustering that keep your Parquet files nicely sized and indexed. Their AI features are terrible and I can't tell you how many Zoom calls I've joined to rescue people from wild goose chases their model took them on.
Their main competitor is Snowflake, which imo has nicer UI/UX and snappier SQL execution, but the trade off is that Snowflake has a bit more lock-in, whereas Databricks is easier to kind of just build on top of your existing infrastructure. Snowflake's AI features are also much better in my experience.
>I feel like we might have some a tech theranos on our hands here.
Based on what I've seen some customers paying them, they are absolutely making heaps of money. A "small" SQL warehouse cluster is $8.40 an hour (~$73k a year if you leave it on) and the default size of X-Large is $56 an hour or $490k a year. It goes up to 5X-Large at $729 an hour or $6.3M a year. So they do absolutely let people with horribly optimized data just pay their way out of dealing with the problem, which is something I've seen multiple times.