On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.
This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot.
They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests
here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes)
usage 1: (simple html page of all results)
echo https://www.theatlantic.com/article/619320/|1.sh >1.htm
firefox ./1.htm
usage 2: (retrieve last result)
echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm
firefox ./2.htm
#!/bin/sh
read x0;
x1=web.archive.org;
curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \
|case $# in :)
;;0)( printf "<h2> $x0</h2><ol><pre>\n";
sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
printf "</ol></pre><br>\n" )
;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p);
esac
maybe it is faster than curl, maybe not; you be the judge
#!/bin/sh
read x0;
x1=web.archive.org;
printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms
\nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b
\nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
|exec haproxy -D -f /dev/stdin;
printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
\r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \
|case $# in :)
;;0)( printf "<h2> $x0</h2><ol><pre>\n";
sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
printf "</ol></pre><br>\n" )
;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \
$(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
|exec nc -vvn 127.21 80;
esac;
if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;ficurl version
#!/bin/sh
read x0;
x1=web.archive.org;
curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :)
;;0)( printf "<h2> $x0</h2><ol><pre>\n";
sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
printf "</ol></pre><br>\n" )
;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \
|exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
|exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;";
esac
#!/bin/sh
read x0;
x1=web.archive.org;
printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms
\nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b
\nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
|exec haproxy -D -f /dev/stdin;
printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
\r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :)
;;0)( printf "<h2> $x0</h2><ol><pre>\n";
sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
printf "</ol></pre><br>\n" )
;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \
$(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
|exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \
|exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
|exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"
esac;
if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fiRe slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February:
https://archive.org/details/jonah-edwards-presentation
and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week:
news sites should really be mandated to keep previous versions of their newsstories with all the edits, especially the ones paid by taxes.
https://www.4chan.org/robots.txt
Also, I let a domain of mine expire and the new domain owner (which just plastered ads) had a robots.txt that retroactively removed my “previously archived website” from the Wayback Machine.