Hacker News on BigQuery: Now with daily updates. Top domains and time to post?
medium.com
medium.com
1. The algorithm changed recently. This post uses >40pts as a proxy for front pageness. That's too conservative; even my 10pt threshold back then was conservative. With recent algorithm changes to Hacker News (<1 yr), I've seen posts with 3pts get into the Top 10 for whatever reason, which breaks predictive analysis.
2) The dataset/this submission only includes submissions/ submission scores; comment scores were removed from the API which is disappointing.
3) Given that HN titles/links can be edited by moderators (and they do a good job), it's harder to judge initial submissions from the final result.
4) Slight edge case in the article, but link shorteners are auto-killed which is why youtu.be/goo.gl links are not prominent.
Replying to the comments:
1) Yes, it doesn't take many votes to reach the front page, but once a post gets established there I expect it to gather more upvotes (then crossing the 40 points threshold). 85% of posts on the front page right now have >30 upvotes now, and I expect them to keep going up.
2) We don't have comment scores, but we do have them ranked. So you could assign them points based on what position they are relative to others, and what position does their parent have. Looking forwards to the experiment :).
3) Yup, titles matter a lot, and unfortunately we don't know what titles were modified after the fact.
4) Thanks for the clarification!
Thinking about 2) - as you comment a lot and you can see your own comment scores, we could train a model that goes from rank/time to score.
Lessons:
- Maybe don't post on a Friday afternoon.
- Maybe don't post from medium.com (low chances).
- >40 score choice seems sound, to look at posts with front-page permanence.
Another reason why analysis is tricky, as such submissions are hard to algorithmically identify.
If manual penalties were public, they could be accounted for in a model. That's another reason why I like Reddit data better; fewer unknown penalties.
I put something similar together (for HN data). One hitch is tracking historical changes as I don't think you can get a raw feed of it so stuck with polling a historical range to ensure you get the latest version of a comment.