Hacker News Reader

How does ibis manage ordering of the data frame for SQL based engine?

One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.

View on HN

this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data

so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up

Reply on news.ycombinator.com