Hacker News Reader

Comment by stuxnet79 | Hacker News Reader

> (Yes, my dear aspring data scientist, do not load everything in a huge DataFrame, go learn yourself some SQL :-) )

This hits a little bit too close to home for me. I am quite proficient at writing performant SQL queries and recently started using Pandas. I find the data frame abstraction better for certain data manipulation tasks. Assuming there is enough RAM available is it still better to offload everything to the database engine?

View on HN

As usual: it depends... If you're doing prototyping and are working in a Jupyter notebook, sure, go ahead and work on the Pandas-level. Once however you're done with prototyping and have settled to a "final_df" (I bet you have something like that in your last notebook cells), maybe you should think transforming some of the "columns" to sql queries (which are VCS-able, sometimes are faster, and most importantly other people can use them too. And instead of 10 people loading 10 different DFs, you can have 10 people querying the same table/view.

> Assuming there is enough RAM available

You will also have to assume you don't care much about the latency introduced by transferring everything over to your working process.

(And in batch data processing situations you typically don't care much.)

Dask has an out-of-memory dataframe implementation. Works great! I think it might support sql queries for that matter.

Dplyr works great with SQL (both SQLite and others).

Reply on news.ycombinator.com