Hacker News Reader

Hacker News

Comment by Anon84 | Hacker News Reader

Full thread

Anon84

I was hoping for a dump of the actual database...

View on HN

There's a torrent link to it in the article, or do you mean something else?

There could be an official detailed api.

It uses ordinary files, not a database.

I meant the actual YC/PG Hacker News database, not one reverse engineered from the HTML output.

What's the difference? Other than parsing issues?

Take a look at the example in the page:

    <HackerNews>
    <row>
        <ID>1</ID> 
        <Url>http://ycombinator.com</Url> 
        <Title>Y Combinator</Title> 
        <Text /> 
        <Username>pg</Username> 
        <Points>39</Points> 
        <Type>1</Type> 
        <Timestamp>2006-10-09T20:33:12.700</Timestamp> 
        <CommentCount>15</CommentCount> 
    </row>
    </HackerNews>

If you really want to understand the HN community (or online communities in general) this data doesn't tell you much. There are many interesting questions this data cannot answer.

When was each of those 39 points gained?

Who upvoted?

How many points did the submission have when each comment was posted?

How does the number of points affect the number of comments?

Are there cascades of votes (several users voting in quick succession)?

For each of the comments, who voted up and who voted down?

I don't think the actual HN "database" has that much info. You would need to reconstruct this from the complete server logs, along with application logs for PG's direct updates (assuming he doesn't use the web interface)

I imagine it does... It has to at least know who upvoted, to prevent revotes. The timestamp is frequently stored as well.

It has to at least know who upvoted, to prevent revotes.

HN has to know who voted, not which way someone voted. HN doesn't let you change your vote once its been made, but it does let you downvote once you have enough karma.

Maybe they use a bloom filter to store who voted per comment, that would be an efficient way to prevent upvotes without needing to store much data.

I don't think this would work well. The occasional false-positives would look like annoying glitches, every now and then making it look like your account had been hijacked by hiding the buttons as if you had already 'voted'. Although maybe I'm not getting the math right?

Also, the existence of the 'saved stories' link from the user page would seem to indicate that for stories at least a full list is being stored.

I think it would have been neat if someone created a browser-extension that let users contribute such data, but I guess preventing abuse could prove to be an impossible task, since there is no way that I can think of for a third-party to check whether the user really performed an action without having to perform those actions for the user, in which case we would have to trust our login-details to that third-party.

Timestamps and an accurate karma count (instead of the -4 cap) spring to mind. I'm sure there are other issues as well.

Reply on news.ycombinator.com