ArchiveTeam is downloading and saving content from Blogger
wiki.archiveteam.org
wiki.archiveteam.org
> No updates to Blogger have been documented since September 2020. Google has also moved away from Blogger for their own company blogs. For these reasons, Blogger may be at risk of shutting down.
> Archive Team did a discovery between February and May 2015, but did not begin downloading actual content until November 2023.
https://www.reuters.com/technology/google-delete-inactive-ac...
If the format is nameofblog.domain.tld, then to get a JSON feed of all the posts, something like
curl -sA "" https://www.blogger.com/feeds/nameofblog/posts/default
I only managed to get a subset, but I might not have found the right example.
(I wasn't sure where to get nameofblog from)
I don't have any particular blog in mind, I just want to see how big an XML file that old platform is willing to serve me. I'm interested in streaming large responses generally, see https://simonwillison.net/2021/Jun/25/streaming-large-api-re...
Can you provide an example URL that I can run curl against to get the full content of a blog?
Been some years since I last used this.
blogID is extracted from HTML at nameofblog.blogspot.com as in the example script below.
According to personal preference, I have a custom filter to process the XML into formatted plaintext, no images.
To see some sample blogNames and blogIDs, search
site:www.blogger.com targetBlogID
Something like
tr @ '\15' << eof|openssl s_client -tls1_3 -connect 142.250.185.132:443 -ign_eof > 1.htm
GET /search?q=site%3Awww.blogger.com%20targetBlogID&num=100&filter=0&start=1&scoring=d HTTP/1.0@
Host: www.google.com@
Connection: close@
@
eof
firefox ./1.htm
test $1||exec echo usage: $0 nameofblog
x=$(curl -sA "" https://$1.blogspot.com|grep -o targetBlogID=[0-9]*|sed -n '1s/.*=//p;s/ //');
test ${#x} = 19||exit;curl -sA "" https://www.blogger.com/feeds/$x/posts/default;It seems trivial to poison the uploaded content. And there's no way to validate that the content has not been tampered with in flight, unless a redundant fetch were performed in a trusted environment, at which point farming out the work serves no purpose.
Secondly, this doesn't solve the discoverability problem. Sure, after Blogger shuts down, a particular webpage on a blog might be preserved. But you will only be able to view its content 10 years in the future if you happen to already know the exact URL where it used to exist.
Isn't a potentially-tampered copy better than none at all?
I guess it depends on the specific nature of the tampering, but I can think of quite a few situations where no copy is indeed better.
The problem I had with ArchiveTeam is that they are very insular, and don't accept help quickly or easily from someone who is not already inside their inner circle. I'm guessing they've been burned more than a few times.
There are publicly available versions of their tools, but those aren't the versions that actually get deployed. You have to know or be told the private locations for the tools that actually get deployed.
What's important to them is not how much resources you could help bring to bear to solving this problem that you care about right now, but are you still going to be here in a year to help them with the 99th problem that you don't care about?
I can't say that I blame them for being insular in the face of whatever their past experiences were. I just wish that they had let me join and help them, because I do think they have a good mission and they deserve to be helped. But if they won't let you help, then all you can really do is sit on the sidelines and watch. Or not, as you choose.
I chose not to stick around.
But when you genuinely are trying to help it can be pretty frustrating to find a party skeptical or non-responsive. Eventually I turned the project over to someone else. It's still alive ( https://geocities.restorativland.org/ ) and the reocities.com domain still redirects there so at least it wasn't for nothing.
This is amazing, thanks for sharing!
Ironically, the only place the story lives on is at Archive.org :) :
http://web.archive.org/web/20170110032949/http://www.reociti...
Consider also: they're just flawed; they're cliquish and irrational and require you to figure out a way to ingratiate yourself to them ahead-of-time/out-of-band. They don't have to be the victims of past harms.
But I think Occam's Razor tells us that it is more likely that they are victims of past harms, which is why they have become so insular.
In the absence of more authoritative evidence from the parties in question, your guess is as good as mine.
That doesn't follow. That explanation is hardly parsimonious. If anything, Occam's Razor favors an explanation like the one I gave, which is far less complex.
I still think it makes a lot more sense if they have actually perceived harm from others coming in from the outside, which now explains the behaviour we see today.
But as I said, without more authoritative information from the parties in question, your guess is as good as mine.
Asking "what's the reason" to try to explain something irrational is a dead end. There is no reason by definition.
Taking the position that a person themselves is authoritative on matters concerning them leaves you vulnerable to an unreliable narrator.
I think that's where we differ. You see it as a purely irrational behaviour on their part that doesn't need explaining, whereas I see it as a rational response that does actually have one or more trigger events in the past.
They could use some form of quorum, where the archived page is distributed to multiple users, preferably in different countries, and only accepted if the results mostly agree. (Given dynamic pages from some of the archived services, it is an interesting problem as to how you would compare if two pages are almost the same.)
(See the first sentence under the "strategy" section, as well as that they say blogger.com/profile/NUMBER redirects to a blogspot subdomain)
I think those of us that remember the Internet from 20 years ago think of Blogger as the company and CMS and blogspot.com as merely the hosting domain.
Blogger seems to not have been on my radar though I was around then, and is such a generic word that imo it's not a clear name to use when Blogspot is also available and everyone knows it
https://timetravel.mementoweb.org
The asterisk at the end selects all the archived pages of the host:
https://web.archive.org/web/*/https://comediventareilmiocane.blogspot.com*You can contribute to the effort by running these two simple steps on your vps or home network:
docker run -d --name watchtower --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable --cleanup --include-restarting --interval 3600
docker run -d--label=com.centurylinklabs.watchtower.enable=true --restart=unless-stopped atdr.meo.ws/archiveteam/blogger-grab --concurrent 20 hackernews
Important: don't use VPNs, proxies, hotspots, etc.
Is there any way I can contact ArchiveTeam and ask them if they already archived it?
https://web.archive.org/web/20230000000000*/https://plus.goo...
Most of those are focused_crawls which would have been the result of efforts like ArchiveTeam.
Then the full set is here:
Basically, just as testing recovery is part of a backup strategy, testing the usefulness of the “saved” information ought to be part of any “saving” strategy.
Edit: You can try links posted to HN e.g. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... or https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... and see if you can read any of them via the archives.
I've found the cdx search feature of archive.org incredibly helpful to quickly find functioning pages (status code 200) and to see all contents that has been captured under a certain domain or page. For example, you can make a query using Linus's google+ page as the URL parameter with an asterisk as a wildcard[2]. This allows you to see all the captured content from Linus's Google+ profile, including individual posts
1: https://web.archive.org/web/20190331095744/https://plus.goog...
2: https://web.archive.org/cdx/search/cdx?url=https://plus.goog...
It's great that the reality is not "nothing is visible" but rather "a bit is visible" — still, the archive just isn't very useful, compared to the impression from statements like "98.6% of profiles saved" etc. A little bit of forethought or testing its usability would have made a huge difference (some js code to expand posts and comments before saving the page, for instance, as some other archiving scripts at the time did, which I used to save the profiles of myself and some of my friends… didn't run it on popular public profiles thinking that ArchiveTeam would have likely covered it in their 98.6%, but it turns out they did not).
In the future, with the right assistance, I'm sure they could grab better archives in the limited time they have.
I still don't think that makes what they did save less useful or make it less admirable.
It would be like a library burning down and then telling the firefighters that the few scraps of books they managed to save are not useful. Why didn't they use tools needed to save the books better?
1. Re "In the future, with the right assistance…", they don't seem to be very open to assistance, as mentioned in another comment (https://news.ycombinator.com/item?id=38411567)
2. At the time they were saying things like 98% of content saved, which led one to believe they'd get to properly archiving some of the popular profiles at least, but it turns out everything is only very partial: in your analogy it's like the firefighters celebrating that they saved most of the books, but it turns out they got only the front cover of each book. (Which is great, and better than nothing! It's just… if they had made it easier to see what they were saving, maybe more people would have been able to help? Or would have saved the ones they particularly wanted?)
You simply have to click the link to the full post. To browse all archived direct posts, you can use this URL:
<https://web.archive.org/web/*/https://plus.google.com/%2BLin...>
Maybe we invent archiveDNS which maps urls -> archive urls.
Still good to have it around for those who want to do the deep dive.
If there were were a sufficient number of popular Blogger blogs, that could carry the financial burden of the unpopular ones, but the platform seems to be stagnant and unpopular, so that model doesn't seem to be viable either.
I'm not sure what went wrong, Blogger had good market share years ago!
Ghost's public, live financial metrics are here: https://ghost.org/about/
I agree Google has done almost nothing with Blogger. I’m a bit surprised it wasn’t shut down long ago.
They did shutdown the subscribe-to-posts via email feature, which was like a free unlimited subscriber newsletter.
I don't even want to get into why Blogger is so great, because if I say what I know, people will start abusing it.
Such as? I'm not aware of any.
> I don't even want to get into why Blogger is so great, because if I say what I know, people will start abusing it.
Unlimited uploads? Sharing full pirated movies on Blogger?
If you're into photography, you've probably heard of it.
Medium it's so bloated and clickbaity, that I would take an old regular blog in WordPress or blogspot despite being more ugly. A lot of technical documentation of some niches, is still only present in blogs of this kind.
(Full disclosure, I'm a paying user of mataroa, but no other relationship to either of those sites.)
I wonder if they made a big list of the old content that would get deleted under their policy, and decided that some of that actually still brings in enough ad revenue to be worth keeping. And it was easier to just keep everything than make carve-outs to the deletion policy?
https://www.cnet.com/tech/services-and-software/your-old-goo...
By end of 2025, ArchiveTeam should also archive Flutter SDKs. I don't have a good feeling about it.
if i was born earlier or later, those would've translated into web dev skills, and i'd be a dev much sooner.
i'll try to recover my old gmail to check if my blogger sites are still okay.