Hacker News Reader

Comment by minimaxir | Hacker News Reader

While data.table does have impressive benchmarks, the rise of dplyr in the years since with its significantly more readable and concise syntax makes life a lot easier for data sets that are not super large. (https://cran.rstudio.com/web/packages/dplyr/vignettes/introd...)

The vignette linked above describes some compatability between dplyr syntax and data.table, but I admit I have not tested that.

Nowadays, if you are working with multigigabyte datasets, it may be worth looking into Spark/SparkR (https://spark.apache.org/docs/latest/sparkr.html) for manipulating big data in a scalable manner instead, as there is feature parity with data.table + analytical tools [GLM] as a bonus. (The syntax is still messy, to my annoyance)

View on HN

You can rent an r3.8xlarge instance with 344GB RAM on AWS for $2.66/hr. If that isn't enough, an x1.32xlarge instance with ~2TB RAM is $13.34/hr. Assuming your data is already in AWS, of course. Of course those machines have a crap ton of CPU power (I'm explicitly not saying cores because I'm mystified by what vCPU really means) as well which you will be hard pressed to take advantage of but if it's RAM you need they're super-cheap.

If you prefer Azure, a G5 instance with 448GB RAM is available for $9.65/hr.

Not a lot of need for Spark/SparkR for the vast majority of data sets given the cheapness of compute these days.

vCPU is a hyperthread

The CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scripts with the help of foreach[1] and doFuture[2].

Such an approach worked to great effect for me recently, when I needed to perform a `zoo::rollapply`[3] across a time series with tens of millions of rows. The speedup when throwing more cores (7, on my laptop) at it is roughly linear. If I ever need to scale up the analysis to hundreds of millions of rows, the 128 vCPUs of an x1 EC2 instance would be well worth the $$/hour.

[1] https://cran.r-project.org/web/packages/foreach/index.html

[2] https://cran.r-project.org/web/packages/doFuture/index.html

[3] http://www.rdocumentation.org/topics/1505094

Shameless self promotion: my `slide_apply` function shows 5-10x improvement over `zoo::rollapply` (despite no Rcpp call, somehow...)

https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee

if your function has a well-defined rolling form, e.g. rolling mean or standard deviation, you should use a pre-optimized function for that. package `catools` has a bunch of useful ones.

Thanks! As it happens, my function doesn't conform to any pre-optimized facilities, though I wish it did. At some point, I may look into writing an optimized variation, with the help of Rcpp, or even using the (inspiring!) fortran approach of quantmod/xts.

The syntax is very concise and cryptic, but what is messy about dt[i,j,k]?

data.table transformed the way I do analysis. Honestly probably 75% of my lines have are dt[] calls. It allows me to analyze on 500M row data.sets routinely on a powerful workstation interactively.

How much memory does a 500m row data set use in data.table?

While I like and use dplyr time to time, I wish it could handle POSIXlt datetime. Also, it seems to barf on some dataframes just because the df contained a column format, dplyr doesn't support even if it is not being used in processing.

just use lubridate, it has a lot less headache to it.

my beef is with the new tibble-ish format of dplyr. the format doesn't support lists as cells, unlike data frames. right now, `tbl_df` is a good in-between, but since dplyr is converging to the tibble/feather format, it's gonna be rough

Nowadays I almost always use both data.table and dplyr together in my projects. It's a really expressive combination.

Do you use data.table syntax, or do you use dplyr with the tbl_dt class to abstract that away?

I was a data.table acolyte for years, but after being forced to learn dplyr I can't imagine going back. I'm hoping if I need update-by-reference in the future I can use tbl_dt but I haven't played with it yet.

If I'm using them together, I'll start off a pipeline with a big DT expression, then pipe it through a few dplyr verbs as a kind of "post processing."

I find that some tasks are frustrating with one package but easy with the other, so more frequently I just have some "dplyr lines" and some "data.table lines" in my scripts.

And the compatibility between the two has improved a lot, so you don't get so many "invalid selref" warnings and unexpected behavior when you combine them in pipelines

Reply on news.ycombinator.com