Network News Transfer Protocol (NNTP) (2006) | Hacker News Reader

Network News Transfer Protocol (NNTP) (2006)

tools.ietf.org

The spec isn’t so interesting as what is needed to implement it efficiently. Maybe less of a challenge today since usenet is out of vogue. I knew someone who ran a Usenet service in the heyday and they spent a lot of time working around hotspots and limitations in storage capacity and I/O throughput. They typically had clients and servers accessing the same servers and it could get overwhelming during peak hours. I think all of these things are solved problems today, if someone was implementing an nntp server from scratch today I expect the backend would look very different - probably a lot more separation of content and metadata, a lot more caching between front end clients and the backends. Having SSDs instead of spinning disks would make a huge difference.

Usenet is low key and mainly for file distribution nowadays but it still has its niche.

I do like to try implementing the protocol for making servers whenever I'm trying to learn a new language, it's my Hello World. I've never produced anything that works to any significant extent because unlike HTTP there aren't a huge number of implementations to learn from and the protocol is a bit vague in parts. It's an arcane topic and not many people to ask.

Oh yes, the wonders of downloading binaries from Usenet. I always wondered if people left out a part on purpose, it seemed suspicious to me that out of n dozen pieces of a file that arrived, one piece consistently would not. About as bad as the people who seed a file except for the last block.

Surely it's more evil to seed a file except for the first block?

This is part of the reason .par files are popular - as long as you can access a fixed number of par files, you can recreate the source file.

> Usenet is low key and mainly for file distribution nowadays but it still has its niche.

I always find it funny that people think of Usenet "mainly for file distribution" (though that happened), when I used it exclusively for discussion... long before the News servers had the capacity to deal with all those alt.binaries.* groups.

> I knew someone who ran a Usenet service in the heyday and they spent a lot of time working around hotspots and limitations in storage capacity and I/O throughput.

A large portion of the challenge is the alt.binaries.* tree. The Big 8 (comp.*, rec.*, etc) is probably relatively easier:

* https://en.wikipedia.org/wiki/Big_8_(Usenet)

* https://en.wikipedia.org/wiki/Usenet_newsgroup#Hierarchies

The advantage of NNTP over IMAP is that your database logic is actually very simple (it's essentially an append-only list, although you do need also need a hashtable for message IDs), and your I/O bottleneck is largely going to be on what is effectively a key-value store for the actual message bodies. This is contrast to IMAP, which is essentially a database synchronization protocol that wasn't designed as such, and thus requires the server to keep track of a lot of extra information to potentially keep clients in sync.

Reply on news.ycombinator.com