Hacker News Reader

Comment by padthai | Hacker News Reader

All these articles always say that Python has better preprocessing compared with R. Where? I find tidyverse/data.table much more elegant than pandas, scipy et al. The only thing that I like more in Python it is how it handles streams/generators.

View on HN

I use both R and Python extensively, and also was confused for a long time why the conventional wisdom is that one of Python's strengths is a data cleaning and preprocessing. I too always found data cleaning to actually be one of R's strongest features. My hypothesis: I think there is an inconsistency between what is meant by "cleaning/processing" between stats- and CS-oriented people. To the latter, it often just means turning unstructured or tree-like data (JSON/XML) into a tabular format. As a statistician, I think of it being a much larger set of operations; specifically those supported by packages such as dplyr, tidyr, and stringr... I'm not sure if that interpretation is correct, but Python does excel in the XML/JSON => CSV step and R is great at all the processing that happens after that.

Ya, completely agree with this (having also used both extensively). dplyr can also connect to remote SQL server so data don't have to be local. Maybe pandas does this now too but in my experience SQL connections were generally more painful in python

This is what SQLAlchemy is for. It's one of the best ORMs out there and makes interacting with a SQL server a breeze.

The last time I preprocessed with Python I only used csvreader. Client had sent a tree structure as two column table, so in one pass I outputted SQL for the recursive table with a bit of if-second-column-empty-then-figure-out-depth-of-first-column logic

Maybe R would work, but I'm familiar with Python. So Python stayed out of my way

Reply on news.ycombinator.com