Hacker News Reader

Comment by dependenttypes | Hacker News Reader

What would be a good free NNTP server or NNTP archive?

The D programming language forums work as a NNTP server as well as web forums. I have in the past downloaded all content from the forum allowing me to have fully offline archives of threads. This is so underrated. I think NNTP could make forums much more superior although it feels like there arent many clients springing up AFAICT.

Adding some new NNTP features to Thunderbird was my introduction to open-source software and ultimately led me to being one of the primary maintainers.

NNTP is a wonderful protocol, arguably the simplest of the 4 mailnews protocols (IMAP, POP, SMTP, and NNTP). While it seems to share the same basic format as RFC822 messages, it actually tends to avoid some of the more inane issues with the RFC822 formatting (generally prohibiting comments and whitespace folding).

Unfortunately, the internet by the early 2000s started turning more and more into an HTTP(S)-only zone. Usenet itself hemorrhaged its population base, especially as ISPs shut down their instances (e.g., because someone found one child porn instance somewhere in alt.binaries.*).

We periodically hear calls to replace the D NNTP forums with "modern" forum software, but naaah, nobody does it better than NNTP!

Vladimir Panteleev did, though, write a web interface to NNTP:

https://forum.dlang.org/

which is also freely available:

https://github.com/CyberShadow/DFeed

The only things I would ever modernize about the D forums if I were to ever bother is just CSS style or something, but honestly, they work, they're faster than most forums out there and aren't crashing all the time (I'm looking at you vBulletin!) so it's fantastic.

I'm a broken record on this, so you may have seen me point it out before, but NNTP started dying in the mid-late 1990s, when binaries took over. It was extraordinarily difficult to keep reliable full-feed binaries (NNTP is the dumbest conceivable way to share large binaries), and if you couldn't do that, customers would yell and ultimately abandon your service for a cheaper one, while opting for more centralized Internet NNTP services.

Ultimately I think the web would have eaten Usenet anyways, but it's a shame; we were Freenix-competitive (I think I independently invented the INN history cache), and that was some of the most fun I've had doing systems engineering work.

> I'm a broken record on this, so you may have seen me point it out before, but NNTP started dying in the mid-late 1990s,

I didn't start posting to usenet before 1999 and was a regular poster in a few groups from that time up till around 2014. Excluding spam, what was the activity level of groups before and after eternal september?

> It was extraordinarily difficult to keep reliable full-feed binaries

I don't understand why ISPs wouldn't just limit their newsfeed to the text only newsgroups? Did peering arrangments require one to also provide binary newsgroup access? IME, ISP and university news servers had mediocre binary completion rates at best. If someone wanted binaries, they could always subscribe to one of the paid newsfeeds that provided better completion. So I don't really see the incentive for providing binary access at all at an ISP/educational institution level.

That was exactly what Universities where doing here in the 90's. We didn't have physical disk space or bandwidth for a complete feed, but where able to provide all the non-binary groups in the main top levels (including alt.*).

The ISPs needed the binaries though, because that was all they were used for. People read the text on their Uni systems, where they got free dialup, rather than chew up their ISP connection time quotas.

> People read the text on their Uni systems, where they got free dialup, rather than chew up their ISP connection time quotas.

At dialup speeds, the only practical binaries one could download would be images. mp3 files were practically the upper limit of file size one could download. Beyond that, articles would expire off the server before they could be downloaded.

Without broadband and better completion rates, which commercial ISPs didn't really provide (especially the latter), customers probably wouldn't really try using their usenet feed for that purpose when they had other alternatives for binaries.

I just used my ISP's usenet access for text groups until they discontinued it.

If you disabled binaries, customers would get angrier than if you simply didn't provide Usenet at all. Nobody that cared about Usenet would sign up to a provider that didn't provide full feeds. I agree that it's irrational, but it also destroyed Usenet, a couple years earlier than I think it would have otherwise.

For what it's worth, I never called my ISP's customer support to complain about bad completion rates on Usenet. Logically, people would have found other ways to download what they wanted, either by finding an alternate source, or another usenet feed to replace or combine with their ISP's.

What really took down usenet was when Andrew Cuomo, back when he was the state attorney general of New York, made a deal with several major ISPs to restrict access to child porn via usenet.

This lead to many ISPs discontinuing their usenet service, which in turn lead decreased the number of people posting to text groups. Within a few years of that happening, practically all the regular posters in the groups I used to frequent just stopped posting. Those same groups now only have spam posted every several weeks based on what I've seen via google groups. Prior to that, these groups had plenty of active discussions going back to the mid '90s and earlier.

I was the tech lead at the most popular ISP in Chicago in the mid-late 1990s and I assure you that people complained, on Usenet, in email, and in phone calls. And we kept a full feed!

Ah I see, I'm not more curious that you say it's really simple, I haven't read the spec much personally. I loved the concept of the D forums so much I intended to attempt to setup my own NNTP daemon from scratch, but it's in a bucket list of projects I want to try out, the only resource I could think of reading are the RFCs not sure if anybody else has documented Usenet otherwise.

What ever happened to grouplens? That is a protocol/system for collaborative filtering that used Usenet as a guinea pig.

Probably a bad idea, according to what we know now about echo chambers in social networking.

http://ccs.mit.edu/papers/CCSWP165.html

What you can do with NNTP is run a local NNTP caching server. Then connect to that server instead of the real one. Your caching server can retain articles as long as you want; much longer than the upstream server.

(Though mere long article retention is not necessarily the best archive interface, of course.)

Disclaimer: I'm not well-versed in the solutions in this space. Maybe there is some NNTP cacher out there that also has a web archive interface into it or whatever.

Yes, and I have 100% of the D newsgroups archived back to the very first post. Anyone can get them from the D NNTP server. I also wrote a program to create static web pages from them:

https://github.com/DigitalMars/ngArchiver

and the generated pages:

https://digitalmars.com/d/archives/digitalmars/D/index.html

When we were working on the history of the D programming language paper, this was an invaluable resource.

Back when I was first learning about D (over 10 years ago), I crawled those archives and sorted all the posts based on the comment count. It revealed many interesting topics and the history of how D progressed as a language.

I wish forum.dlang.org had a quick way of browsing just the top list of the most commented posts.

Well it is open sourced on GitHub you are welcome to make tickets. Maybe thats a small type of enhancement the D forums could benefit from. Filters that produce informative pages / results.

Good idea. Never thought of that!

I've been using the NNTP server provided by https://www.aioe.org/ for quite a few years.

There is also https://www.eternal-september.org/ which I used.

AOIE requires no authentication. The Eternal September server requires account registration via the web site; then you use an authenticated NNTP connection.

There are other servers out there.

These sites do not provide any archive.

Reply on news.ycombinator.com