Ask HN: What is the Hacker News database structure? Does it use tables or NoSQL? | Hacker News Reader

Ask HN: What is the Hacker News database structure? Does it use tables or NoSQL?

HN always seems pretty fast and is probably well designed from a data structure standpoint.

It seems that each comment and submissions are assigned a monotonically increasing number; do both submissions and comments share the same numeric space?

How is the content hierarchy stored? Is it a single large table with (for example) a structure that looks like id, parentID, contenttype, content?

Is data stored in an OLAP SQL database, on disk as flat files, in DBM files, etc?

What kind of performance is required? Is the database distributed across more than one machine?

View on HN

The old version in arc, mirrored at https://github.com/wting/hackernews/blob/5a3296417d23d1ecc90..., uses the file system as a database.

https://github.com/wting/hackernews/blob/5a3296417d23d1ecc90... shows the monotonically increasing number:

  (def new-item-id ()
    (evtil (++ maxid*) [~file-exists (+ storydir* _)]))

It's used for stories:

  (def create-story (url title text user ip)
    (newslog ip user 'create url (list title))
    (let s (inst 'item 'type 'story 'id (new-item-id) 
                       'url url 'title title 'text text 'by user 'ip ip)

and comments:

  (def create-comment (parent text user ip)
    (newslog ip user 'comment (parent 'id))
    (let c (inst 'item 'type 'comment 'id (new-item-id)
                       'text text 'parent parent!id 'by user 'ip ip)

No clue how that code structure compares to current implementation. For all I know it's completely different.

I believe it is still largely filesystem based, so NoSQL?

Thanks! That is really interesting. I wonder if it still uses the filesystem as a database (I often do that myself, especially for rarely-read or config data.)

The filesystem is an excellent database if your queries can be expressed as a path. It's literally what they're made for.. A system for filing information.

As things are written to file, are they pushed to some cache? I’d imagine we are not reading directly from a disk.

Is each HN thread it’s own file? How do we add additional posts into the discussion, maintaining the hierarchy? If each file is a thread, then adding to the discussion means we deserialize the file into memory into some data structure and perform updates in memory? Or is there some syntax that allows us to append to a specific blob/container in the file?

Another way could be maybe you have a file and folder per HN thread, and each response and it’s own responses are their own folder and files (recursively). Not sure how performant this would be though.

Reply on news.ycombinator.com