As a potential user of Ibis, how do I know they won't eventually drop support for other backends until it just becomes a frontend for DuckDB?
1. The value prop of Ibis is to avoid vendor lock-in and allow users to more easily swap out execution engines that meet their needs. So it's valuable to the project to have "competing" backends supported (e.g. DuckDB, DataFusion, and Polars for local, and Snowflake, ClickHouse, PySpark, Databricks, etc. for distributed) -- the idea is a standard like Apache Arrow that everyone adopts to stop this cycle of tightly coupling a dataframe interface to an execution engine 2. Ibis is primarily supported by Voltron Data (my employer) which is itself a vendor, so the real worry would be that Ibis drops all backends until it only support Voltron Data's engine (Theseus)... 3. ... but Voltron Data cannot do that for a number of reasons. First, Ibis is an independently governed open source project. Second, it would not make any business sense. You can read some more on that here: https://ibis-project.org/posts/why-voda-supports-ibis.
The general idea is to add more backends, not remove them, but pandas is a special case where it introduces a lot of technical complexity with (arguably) negative benefit to potential users -- the DuckDB backend can do everything the pandas backend can do, but better
Maybe this should be a focus. Putting together killer demos that show this to people. Because I don’t see it.
Maybe I’m thinking about it in the wrong terms. I think of DuckDB in terms of what we’re doing with pandas (which isn’t perfect but works), and also as an embedded database like a SQLite alternative.
In both aspects it seems like we have more friction working in DuckDB than in pandas and SQLite. Most common problem is getting errors about the DuckDB file being locked or in use, by the process that’s trying to read from it (which just spawned). I thought maybe it was me, but even Jetbrains tooling had the same weird issues with DuckDB.
Compared to SQLite, which just works, with virtually any language or tooling, I don’t see the benefit to DuckDB. I see some people say it’s better and more efficient for larger data, but I couldn’t get it to work reliably with tiny data on a single machine.
DuckDB is much nicer in terms of types, built in functions, and syntax extensions for analytical queries and also happens to be faster for big analytical queries although most of our data is small enough that it doesn't make a big difference over SQLite (although it still is a big improvement over using Pandas even for small data). DuckDB only just released versions with backwards compatible data storage though, so we don't yet use it as a source of truth. DuckDB has really improved in general over the last couple years as well and finally hit 1.0 3 months ago. So depending what version you tried it out on tiny data, it may be better now. It's also possible to use DuckDB to read and write from SQLite if you're concerned about interop and long term storage although I haven't done that myself so don't know what the rough edges are.
You don’t. Follow the market share. That’s as the close as you’ll get to a guarantee.
Ibis announcing they’re dropping support for pandas would be like Astro announcing they’re dropping support for React.
It only means fewer people adopt the new thing, not that the thing with the most market share dies off (looking at you, PHP and jQuery).
This is extremely misleading and a false analogy. Read the blog post.
If you still come to the same conclusion after reading the blog post, feel free to reach out on Zulip, more than happy to chat about it.