Options for Saving Yahoo Groups Content | Hacker News Reader

Options for Saving Yahoo Groups Content

researchbuzz.me

71 pointselorant31 comments

I wrote up some instructions on using YahooGroups-Archiver and dealing with the kinds of failures I've seen with it: https://www.jefftk.com/p/archiving-yahoo-groups

> They're not that wasy to read, because it doesn't do any kind of quote folding, but we can always do that later.

I can create an endpoint to bulk import/process content to FWD:Everyone if there is enough interest. It's sort of the opposite of an archival product, where it's designed to aggressively reformat content to improve readability.

E.g. here is a comparison between a thread in Gmail and a thread that's been formatted with our tech:

https://www.youtube.com/watch?v=zAXsHiqQm1E

https://www.prettyfwd.com/t/Imac8ycyQe26vDZd8pEkMg

looks like the Archive Team is on it: https://archiveteam.org/index.php?title=Yahoo!_Groups

This is mentioned in the article so... yeah. Anyways AT will not get everything. It's hard to archive private groups, rate limiting seems unpredictable, bugs are still getting worked out of the scripts. If there's a group you really want saved, don't sit around and hope AT gets it. Use the tools in the article.

I had experience working with ArchiveTeam on the G+ shutdown.

1. They're AMAZING.

2. They're not superhuman.

3. Major archival projects take time to prepare, set up, and execute.

4. Only public content is even potentially accessible. Much can get lost.

AT did a stellar job with G+, it's still a far cry from the access that was provided when Google hosted the content.

Google actively and by all appearances deliberately thwarted and frustrated archival efforts. This was a tremendous disappointment. It's quite likely Yahoo will do similarly.

If you want your own control over archives, you're going to want to use / run tools directly yourself.

TL;DR: AT is a huge benefit, but it's not a total solution.

Just a question regarding comment format

Shouldn't the TL;DR: message be at the top of the comment, thereby not having to read the entire comment (since its too long :-)) to get the summary statement

This is the one I've been using to good effect: https://github.com/IgnoredAmbience/yahoo-group-archiver

no Mongo database required, just plain old .eml files

You do need to pull some cookie values out for the authentication to work, and the Readme is pretty sparse, but it does work.

Yes, it works great. Tested on a group with 9000 messages.

oops, edit, Actually, I used this one: https://github.com/philpem/yahoo-group-archiver

Yup. I've used this one successfully with messages, message attachments, files, and photos. It works.

That said, on yahoo's side, lots of message attachments the message meta-data claim exists don't actually exist anymore because yahoo has lost or deleted them. Same with some of the photos. But all the messages and all the files are always there.

I'm about seven months out of having gone through a similar process with Google+. What I'd contributed specifically was in trying to create some structure and process for people and organisations hoping to migrate both content and communities from G+. Success was mixed, but it was, I think, better for what we'd managed to do.

A large part of that was setting up informational resources with an eye for future migrations to use, learn from, and adapt these. In particular:

https://social.antefriguserat.de is a wiki focused (principally) on the G+ migration, but with some generalisable principles. Please feel welcome to visit and add to it. (I'm the principle editor/admin.)

One of the huge wins we had was a third-party end-user tool for data export. I'm checking to see if there's a similar Yahoo Groups product.

https://old.reddit.com/r/plexodus is the "Google Plus Exodus" but wink-wink "public liberation exodus" subreddit for discussion. I've already posted several items on the Yahoo Groups shutdown. Again, for an out-of-band regrouping channel, it's an option for Yahoo Groups users.

I also need to follow up with EFF and some other groups I'd talked to briefly about the issue of migration off public web services.

A number of years ago, I downloaded several Groups I cared about using something like PG Offline (though I don't remember if that's exactly what I used) that gave me an sqlite file. I ended up converting it into MySQL and building a terrible viewer "app" with PHP that just navigated up and down the messages.

I never totally understood what happened to some of my old Yahoo Groups though. Several disappeared completely, and some despite being completely inactive, persisted for years upon years.

What's the reasoning for not working with the internet archive or similar organizations when shutting down a service like Yahoo Groups? It seems that it would be much more efficient to work with them on sharing a cleaned DB dump instead of having multiple groups put lots of unnecessary load on their servers.

Given that they need exporters for individual data anyhow to be GDPR compliant, it shouldn't be a huge project to tune them slightly.

the cynic in me says: why risk legal issues, however small they may or may not be, when doing literally nothing sidesteps that issue? if only 1 user thinks of suing you, legal gets involved, maybe even external attorneys. also, who will pay for possibly hundreds of man-hours of work cleaning up the data?

I don't think that's the cynic in you. That likely is the truth (legal plus labor).

As for cleaning up the data: I don't think that's a large task. Whatever is published now has either been cleaned or is just as problematic from a legal standpoint (possibly more so because it's shared publicly, not shared with a contractually obligated third party).

You're probably right though regarding "not doing anything isn't doing something wrong". But still, saving lots of computing resources, get good PR and friendly news coverage, that has to be worth something.

To a first approximation, there's no PR or friendly news coverage. 99.999% of even the tech world doesn't care. Meanwhile, the effort involved is no one's day job. Someone is far more likely to hear "Why are you wasting your time on this?" than "attaboy!"

Thanks for the perspective, that makes sense to me. I tend to assume that having somebody work on stuff like that is a rounding error for even the smallest department at Yahoo, but as you point out everybody has to justify how they're spending their time to somebody.

I think you are significantly underestimating the amount of work necessary as well as large uncertainty due to potential legal complications.

I assume your downvotes are a case of shooting the messenger because that's absolutely the case. See also open sourcing or otherwise releasing abandonware etc. As a company, there is essentially no upside and there are plenty of downsides including just the time and effort to do even fairly cursory due diligence.

Plenty of companies are more or less willing to just turn a blind eye if someone else archives things without explicit authorization. But that's different from explicitly doing it yourself.

You don't need to be a cynic to think this. Anyone who thinks otherwise is at best naive.

Approach it from the other side:

What's the business value to Yahoo in taking that proactive step?

I agree that there is other value in doing so, but as a business, what value is there? No upside, only risks.

It will encourage users to use your current and new products, if they don't have to worry that you're going to delete all their data.

is there a tool to migrate yahoo groups email conversations to GNU Mailman so that I can search old conversations?

Or maybe create a usenet newsgroup in the alt hierarchy and post the messages there.

Serious: What are you guys saving?

http://tech.groups.yahoo.com/group/concatenative/

... oh, it seems it's already gone.

We should require websites to freeze and archive public content instead of deleting it. We already require media published in nearly every other format to be submitted to Library of Congress, and we should extend those protections to our internet heritage.

>We already require media published in nearly every other format to be submitted to Library of Congress

No. We do not. Registering your copyright provides some advantages if you ever want to sue for copyright infringement. But there is absolutely no requirement to submit materials to the LoC.

https://www.copyright.gov/mandatory/index.html

> All works under copyright protection that are published in the United States are subject to the mandatory deposit provision of the copyright law

You own copyright over works whether you register them or not. To the best of my understanding that means that you are always required to deposit works when you publish them, whether or not you register your copyright. But IANAL, so I could be way off.

EDIT: Here's a real citation[0]

> What is the difference between mandatory deposit and copyright registration?

> ... Optional registration fulfills mandatory deposit requirements.

So you definitely are required to deposit a copy with LoC whenever you publish/distribute in the US

[0] https://www.copyright.gov/help/faq/mandatory_deposit.html

Interesting. I've never heard of that.

So I stand corrected. (Sort of.) Apparently it's a very old provision. [1] It doesn't really make much sense in today's world. Read literally, almost anything you create--including online--must be sent to the LoC.

So, yes, it is a law on the books and maybe mainstream publishers comply with it. But But pretty much no one else does AFAIK.

[1] http://articles.ibpa-online.org/article/need-know-copyright-...

Not necessarily, because it can be debated whether posting something on the internet really counts as publishing, and whether most internet content is original enough to be eligible for protection. A good example of people who are probably violating this law would be webcomic artists.

But I do think the world would be a better place if we added new legislation to require platforms to submit archives of their public content. Perhaps LoC should also be funded to scrape the web to capture smaller sites.

It's one of those thing that's technically required, but that there's no enforcement allowed before a demand is made, and in practice is not enforced.

17 USC 407:

> (a)Except as provided by subsection (c), and subject to the provisions of subsection (e), the owner of copyright or of the exclusive right of publication in a work published in the United States shall deposit, within three months after the date of such publication— (1)two complete copies of the best edition; or (2)if the work is a sound recording, two complete phonorecords of the best edition, together with any printed or other visually perceptible material published with such phonorecords. Neither the deposit requirements of this subsection nor the acquisition provisions of subsection (e) are conditions of copyright protection.

Reply on news.ycombinator.com