laserson

That's directly addressed here: http://www.ibis-project.org/faq.html

My goal was to use mrjob's features, not strip them down for performance. I find mrjob's natural use of JSON very appealing in terms of user-experience. It means that keys can be more complex types without the user having to manually figure out the best way to encode them. I make it clear that this is an appealing property of mrjob, and I make it clear that this is the reason for the slowdown. As is, the code will not work with RawProtocol internally because the key is a tuple of words.

Here is the profiling output: https://gist.github.com/4478737

The input is RawProtocol, which simply splits on tab. But after that, mrjob defaults to using JSON internally, and this is causing a lot of slowdown.

I did only a bit of profiling on mrjob, but it seemed that the ser/de was significantly slowing down the overall computation.

Wow, 3 million "children". There are over 62 million children in the US. That means almost 5% of all kids are getting medicated...

http://www.wolframalpha.com/input/?i=US+population+under+age...

qwerty