The End of Gmane?
lars.ingebrigtsen.no
lars.ingebrigtsen.no
It's sad that the dicks of the world are about to ruin such a great resource. It's also sad that such tremendous goodwill and a large amount of work gets ruined by a bunch of jerks.
Thank you, Lars, for all the hard work over the years.
Personally, I know I've relied on Gmane many times to help solve problems I've had with one open source project or another.
I hope that this resolves well in favor of the protection of this archive as a resource, and I am sad at the recent troubles it has seen.
Lights candles to summon Jason Scott to ingest it into the Internet Archive
I was not aware of this. Do you have a link?
edit: self clarifying my own question. Not sure what the specifics for hosting an AWS Public data set are though.
Big Query
>A public dataset is any dataset that is stored in BigQuery and made available to the general public. This page lists a special group of public datasets that Google BigQuery hosts for you to access and integrate into your applications. Google pays for the storage of these data sets and provides public access to the data via BigQuery. You pay only for the queries that you perform on the data (the first 1 TB per month is free, subject to query pricing details).
[0] https://cloud.google.com/bigquery/public-data/
AWS
> Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications. Learn more about Public Data Sets on AWS and visit the Public Data Sets forum.
The professor I was working with told us about how he lost all his data once with a similar startup attempt. Shortly after I was let go from the startup I pulled all the data via the web service I wrote, luckily before they shut it down.
TL;DR I'd trust an entity like the Internet Archive way over commercial entities, even those that offer free storage for open projects. Sure the Internet Archive could up and go away, but I still trust their motives and their commitment to not let that happen.
The difference would be that if AWS itself wanted to stop providing public archive hosting, they'd just up and do that. Whereas if the Internet Archive was hosting a public archive on AWS, and AWS cut off support for public archives, the Internet Archive would then work to migrate the dataset off AWS.
Non-profits have incentives to do (whatever their charter says they do.) Corporations just have an incentive to make money. If you want a certain "preference function" to carry on into the future without self-modifying into something unrecognizable, you probably want to instantiate that preference-function in the form of a non-profit.
And he has responded. https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/com...
But no matter what, the best possible solution would probably be to move the archive to the Internet Archive, and have that same archive continue to ingest email->nntp and provide access to it.
I get that people fear spammers etc, but at this point, I don't see a way to avoid it if you want a usable web archive.
I suppose that it's one more nail in the coffin of NNTP. I really miss net news, but it probably wasn't possible for it to scale to support the entire world. It's sad, though: for a short time, all the interesting people in the world were able to communicate with one another.
I think Gmane 2.0, if it ever will be born, would have a mechanism of plugins, that allow easy wrapping any web-forum in it, by writing a little template describing the layout of pages like 'this is topic, this is message body' and so on. And the crawler to use the plugins, of course.
Well, except for the mess that is RFC 822. At least NNTP decided to go 8-bit safe eons ago, so you just have to shove everything in UTF-8 and you can screw RFC 2047 in the face.
* RFC 2047 is a pain in the butt
* Handling attachments can be difficult (particularly the inability to safely add binary data)
* The display name/address is both painful in terms of syntax.
* CFWS in general (you can tell who's written an email client by their reaction to the letters CFWS)
* Header fields in general have way too lenient handling, so there's no way to, e.g., guarantee non-ASCII handling (RFC 2047 needs to be handled differently in the Subject: header than from the From: header, and the Content-Type: header does something else entirely).
* Mandatory line-length limits, which means you need to be able to insert line breaks in however you encode custom header values
* No way to get attachments without the body or vice versa
* All metadata needs to be shoved into a header (see above about i18n, line length, and CFWS)
What would you add? Remove?
Why?
For myself: the basic RF282 concept of sender, subject, recipient (or newsgroup), date, and references, plus body, seem reasonable.
How best to take these to present capabilities (and needs) is another question.
NNTP servers were generally provided by your ISP, so ISPs built the capacity out as they added subscribers, and added groups based on the perceived interest. It wasn't especially difficult (and it's pretty trivial today) to offer all the "official" text-based newsgroups with a few days or weeks worth of retention; it only becomes a significant undertaking when you want to offer months of retention for binaries newsgroups.
Similarly, because the NNTP servers were basically a value-added ISP service, they were funded by subscriber fees. You never had a problem with a newsgroup suddenly getting popular and being unable to pay their own hosting fees, or suddenly tossing ads in order to pay the hosting bill, which is unfortunately common with webforums. And you don't have problems with a single server crashing and wiping out the history of a whole group. It's all pretty robust.
Architecturally, netnews was superior to web forums. Where it failed was in not having a good response to the spam problem. I think that had it not been for spam, people wouldn't have left NNTP for HTTP-hosted webforums nearly as quickly. But the decentralized nature of netnews meant there wasn't anyone who could quickly implement any of the myriad solutions that were proposed (cryptographic proof-of-work plus reputation systems probably would have cut down on the worst offenders).
My guess is that eventually, we'll reinvent something that's similar to NNTP but by another name (with the decentralization, server-server replication, gateways to other protocols, etc.), but it's been a long and unfortunate diversion in the meantime.
And given that, you can't really have a store-and-forward edge-caching system that truly resembles NNTP. At most, you can get a hierarchical network like IRC.
Much as I distrust Google for anything these days, 8.8.8.8 / 8.8.4.4 offer a widespread advanced notice of domains tracking up the hit rates, and possible attacks.
Email, as STMP, after decades of nobel service really simply does deserve to die. It's far less about the mechanics of email user interfaces, for which there's a great deal of ink wasted, and much more on privacy -- encryption, metadata, and spam, probably as the three top concerns. Very nearly all the suggested replacements (SMS, various messaging systems, Facebook) have either those problems or those and the fatal condition of being proprietary silos.
The more I reflect on it the more the Web is close to what Usenet was, at least in some regards. Origins post messages, which may or may not die in short order (what's TTL for an Internet page now, a few months on average?). We're increasingly relying on sites such as the Internet Archive to actually provide a view-through-time at content.
What the web (and email, especially in mailing lists) lacks and has always lacked is a requirement to meet standards conformance. DDG "The Web is an error condition" and you'll find a far-too-true-to-be-good rant on that topic.
The loose-standards-anything-will-fly-nearly approach has allowed a great deal of flexibility in page design, but much of that is ultimately painful. I'm finding a much greater appreciation for straight-up HTML 1.0 formatted pages.
A huge part of the problem is exceptionally crappy entity styling defaults. If browsers presented pages sanely (at various resolutions), much of the B-Arc profession of Web design would simply cease to exist. And this would be a wonderful thing.
A simplified markup, acceptance criteria (and rejections for failed formats), cache-and-foreard, client-determined-formatting, security, and either a universal content syndication or UBI type compensation model might address other issues.
I'm getting the sense of some interesting possiblities out there.
Actually, when I DDG this phrase, the only result is your comment in this thread.
> One day, I walked into the break room and heard a coworker say, “The Web is an error condition,” referring to the deplorable state of code out on the Web. I think that was the end of the end for me, because it just depressed me. It depressed me not because it was untrue, but because it was so perfectly true.
> Honestly, I miss the days when Netscape Navigator would just halt rendering in the middle of your page, saying, “No, I will not parse any more of your shit until you fix it.”
> Then IE came out for free. Suddenly, the game of web browsers changed from paid apps to supported by advertising and search revenues. The only way to get users to use your browser (and thus get more money to develop with) was to parse all the shit you used to reject.
> The web became a co-evolution of crap and trying to render crap. (It’s gotten more complicated since then, but because there’s been a habit of rendering crap, no one suddenly wants to stop.)
The Android is also an error condition. Trying to copy/paste and maintain state whilst composing anything complex generally fails.
With web forums you fire a search term into a random crawler and it will likely barf up a url. With NNTP you had to configure the email client to point to the isp server, and then locate a relevant group for what you were interested in. And even then searching the whole archive was troublesome if not impossible.
I watched something similar play out with IRC. Once AJAX allowed random websites to offer chat services non-tech channels basically wasted away, as your random joe and jane just entered some url they were told, or read about in some glossy mag, and presto.
As for reinventing NNTP, i don't think there is any incentive any longer. Especially now that something as big as Reddit can just pay Amazon to scale the back end as large as they need.
Store and forward came to be before the net, and was used during the early net, because of narrow bandwidth.
Yeah, but think of the social scaling issues. You simply can't have every single person interested in e.g. American football reading & commenting on alt.games.football.
News was indeed a far superior architecture & experience to web forums — so long as the userbase was relatively small.
Gene "Spaff" Spafford thinks 50k - 500k users ~1988 - 1994 is probably ballpark accurate. Peter da Silva notes that in 1990 he was the most active user on Usenet ... and under another handle, the 3rd most active. Microsoft did some early 2000s research turning up low-single-digit million distinct user IDs IIRC (I've posted on this previously).
So, yeah, it was small.
Organising people into groups, even ad-hoc, from which exceptionally good messages might be shared to others, might be one way of handling scaling. Clustersizes ranging from a total size within Dunbar ranges, to a 1/9/90 split giving the 10% active users a Dunbar cap (about 300 - 3,000 users per cluster) might work.
I agree that 1990 Usenet in 2016 would be a fatal mistake.
Upvoting. NNTP didn't have it. Reddit does. This allows a reader to see the top posts.
for a while and I don't recall getting any spam, ever.
To make this concrete: put enough people or activity or stuff in some area and you end up with noxious effects. Bulk, bioactive, chemoactive, systemic, and even cultural impacts. For most towns and cities, water pollution and sanitation rapidly become pain points. For transport systems, congestion, but also highway robbery and piracy. For communications systems, it's the annoyance of low-quality or unwanted signals. A key point is that cost reduction makes all hygiene factors worse. You end up with more of the bad stuff. There's also scale effects. Ultimately, you need to apply different cost factors to the wanted and unwanted effects (generally through some form of policing, patrolling, filtering, reputation feedback, etc., etc.).
Or you can limit the total size of unrestricted networks and provide gateways (with filters) between them. And yes, that means you're subject to policies (egress and ingress) on both sides of those filters.
Wouldn't the Internet Archive be interested in archiving all those mailing lists?
Obxkcd: https://what-if.xkcd.com/31/
It sounds like a great opportunity for someone (or a group of people) passionate about Gmane to take over the reins here. I'd hate to see a project with so much useful archived content to disappear.
We have ridden a wave of altruism, and it is unclear if it will last.
Gmane doesn't host the mailing lists itself. Instead it provides access to lists hosted elsewhere. It does that by subscribing to the list and then saving the incoming messages. It then provides access to those messages using NNTP, or various web interfaces (different layouts etc). IIRC it may also provide RSS. It also allows posting to the lists by forwarding the message.
The benefits are great as a reader since this is one place you can to read many lists, and you don't have to subscribe to each one etc. (Some may require subscribing for posting though.)
The other benefit is that if you get people on mailing lists asking why use something archaic like email when the hotness is various web based boards, and the project is immediately and unconditionally doomed because that is what people are used to. You redirect those folks to Gmane where one of the layouts is web board style.
That kind of support is literally priceless, you can't buy it from Microsoft or Google or Oracle, etc. And even when you can get some kind of support from them it takes hours or more likely days and is filtered through multiple layers of semi-competent drones regurgitating the official docs.
I think it's horrible that Lars should be so stressed. All I can think of to say is just hand it over to someone and walk away; you've done your bit and more than most to make the Internet a good place, don't let it, or them, grind you down.
Regardless, hope to see Gmane stick around.
https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/com...
My answer to cope with this is that there may be many different rationalizations in the mind(s) of the attacker(s), or none at all. It is impossible to understand the psychology of trolls. Well, maybe not impossible, but certainly not worth the effort. They are basically noxious pests, like cockroaches in a restaurant or wasps at a barbecue.
Not to take away from the contribution Gmane has made to many of my searches over the years, its web interface was one I always held as the epitome of awful design.
Their content outweighs the drawback, though.
Still, I continue to be impressed by the amount of work people put into something they care about, while the rest of us just expects it to be there. As some one else pointed out, it really makes you wonder what other kind of one person projects we all use all the time.
And yet. If Lars will mail out that disk to somebody, somebody may well just pitch to a bunch of VCs, build a fresh web interface and bury it all in ads.
Couldn't Lars do that himself?
The plastering it all in ads, I mean. As somebody in another comment said correctly, the age of online altruism is dead. We have our ideas who killed it, we mourn the passing of an age, but we can't change it.
(OK, just ignore this if the ads are already all over Gmane. I wouldn't know, I adblock...)
Perhaps he doesn't think the money is worth the admin time involved. He's still have to deal with take down requests and DoS attacks, and also have to deal with advertising companies too on top.
He may also prefer not to see his creation go that way under his watch.
I would suggest a better option would be to throw the content of the disk at a public hosting service if any of those who offer free hosting for public data sets would be relevant, or alternately put it out as a torrent until the content is sufficiently well distributed. I'd be happy to throw a chunk of bandwidth at the initial seeding effort as I'm sure would others.
> the age of online altruism is dead (re: plastering it all in ads)
Just because it is difficult to maintain the old ways, doesn't mean all us old timers are willing to take part in propagating the new ones that we don't like!
> I wouldn't know, I adblock
An example of your "online altruism is dead" fact. "You can pay for it with ads (that I personally won't see so don't expect me to contribute)". As more and more people block ads you have to get more and more crafty to make money out of them which makes the whole business just that little bit more shady with each passing day. As per my first two points I can certainly understand him not wanting to go in that direction, as I'd never volunteer for it myself.
And if if the money would be enough to make the enterprise worth while that is not his main point. He was doing it mainly as a fun side project to help others and it simply isn't fun any more. Maybe it could make enough money to make it worth while in a purely time/income based benefit analysis but he still would rather stop as it isn't enjoyable to him at this stage and if even he wanted to spend that much time/effort to make that money perhaps he has other projects he'd rather spend the time/effort on.
If the service truly could be made a going concern in that way he'll no doubt be inundated with offers to take over and try running it in that fashion, so you'll see the new pay service appear in short order.
Cheap still requires paying money, which is a barrier just for the act. Moreover, there is the fallacy that something free has an infinite value/cost ratio. (fallacious because you should look at value - cost)
The money is almost certainly worth it, but the time involved almost certainly isn't.
Lars wrote a blog post about this, if you're interested.