Hacker News Reader

Comment by retube | Hacker News Reader

Came to say exactly this. The chart seems to start (by default anyway) at the beginning of 2007. I'm guessing the user-base and thus activity of HN has increased considerably since then. So yes it would be good to be able to re-normalise in some way. By the number of registered users would probably be the simplest. By number of comments posted would be another, by number of submissions or even number of words would all be interesting.

Btw - my suggestions got me thinking - I assumed this is looking at posts and comments. Whether just submission titles or if includes comments will affect best way to normalise.

View on HN

It uses submission titles AND comments. How would you suggest going about normalization?

In that case I would do it by number of submissions (posts) + number of comments for each period (3 months I think, looking at your graphs?). I would normalised all quarters to the last complete quarter.

So if say in q1 2011 there were 1m posts + comments, and in q1 2007 10k, multiply the count for q1 2007 by 100.

It would be truly great to have this normalisation optional, e.g option to normalise by comment+post count and maybe also normalise by number of users (could be an interesting comparison). And perhaps have them super-imposed on each other!

Thanks, I will definitely implement the multiplier as you suggested. I like this better than showing as a % of total.

I'm storing the total hits so I will do the calculations on the client-side, which means I just might be able to add some toggles for the normalization schemes ;)

Thinking a little bit more about this, there are of course lots of submissions that don't get voted up or commented on - spam, off-topic, not interesting etc. Maybe you should only use posts that have either at least one upvote or comment, just to filter out the crap.

A first version of this has been implemented. You can't toggle the normalizations just yet, but I figured I should get the fix out there as soon as I could.

Check the "Learn More" section where I give you props for the help :)

Awesome, and thanks :)

Also, see my other comment below for a suggestion re filtering out crappy posts.

Coincidentally, I was also working on an HN Trends app a couple weeks ago (though I got distracted and haven't finished), and mine is normalized (by number of comments). For example: http://hn-trends.heroku.com/trends?q=scala%2Cclojure

Your app looks a lot better, though! (And my dataset ends around Sept. of last year.)

Divide by the weighted average change in the number of hits for the 100 most common English words? Basically, add up all the uses of "and," "or," "a," "the," etc. in each period. If Q3 2009 has 4X the sum of those words as Q1 2007, then divide all Q3 2009 results by 4.

Unfortuantely, many common words seem to be filtered by the API. Some however are not, and they show a common growth pattern: http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here

Reply on news.ycombinator.com