Hacker News Reader

> Build a pipeline like I've discussed in the article, then pipe it into Python or R.

Why not just do it all in Python or R? That way you also get something that will probably work on non-unix platforms.

Over the years I've found that I usually fall into a pattern of starting with low-fidelity automation in languages like shell and slowly re-writing it over time into more higher-level languages, usually python first, then Java. This way, unimportant tasks can be automated in less than 5 min with one of these shell commands. If it breaks or has errors, no big deal. Python works well for figuring out the structure of the solution as an actual program, and then finally a language with static type checking when it really needs to run without errors.

I go to Python first because it's nice to be able to single-step through the script with a debugger and monitor exactly what's happening. I also know Python a lot better than shell script so it saves me a lot of time as well.

Reply on news.ycombinator.com