chrishoyle

Beyond federal websites (.gov, .mil) there are lot of gov contractor websites that are being taken down (presumably at the demand of agencies) that contain a wealth of information and years of project research.

Some below of contractors that work with US AID:

- https://www.edu-links.org/ (taken down)

- https://www.genderlinks.org/ (taken down)

- https://usaidlearninglab.org/ (taken down)

- https://agrilinks.org/ (presumably at risk)

- https://www.climatelinks.org/ (presumably at risk)

- https://biodiversitylinks.org/ (presumably at risk)

I'd love to learn more about what is in scope of the Library Innovation Lab projects. Is it targeting data.gov specifically or all government agency websites?

Given the rapid take downs of websites (cdc, usaid) do you have a prioritization framework for which website pages to prioritize or do you have "comprehensive" coverage of pages (in scope of the project)?

As you allude to, I've been having a hard time learn about what sort of duplicate work might be happening given that there isn't a great "archived coverage" source of truth for government websites (between projects such as End of Term archive, Internet archive, research labs, and independent archivists).

Your open questions are interesting. Content hashes for each page/resource would be a way to do quick comparisons, but I assume you might want to set some threshold to determine how much it's changed vs if it changed?

Is the second question about figuring out how to prioritize valuable stuff behind two depth traversals? (ex data.gov links to another website and that website has a csv download)

Related ongoing discussion

The government information crisis is bigger than you think it is - https://news.ycombinator.com/item?id=42895331

Related ongoing discussion

The government information crisis is bigger than you think it is - https://news.ycombinator.com/item?id=42895331

This administration is taking down public websites and resources at an alarming rate. Including scientific documents, health surveys, project research documents that are referenced daily by government staff, contractors, and US international partners.

I recently became aware of this issue and this post made me interested to know if there are any comprehensive datasets tracking these take downs. Also would be interested to know what other efforts exist to archive at risk government websites.

Partial list of removed pages and entire websites I'm compiling:

- USAID Development Experience Clearinghouse (stores 50yrs of international aid records) https://dec.usaid.gov/dec/

- CDC HIV website https://www.cdc.gov/hiv/causes/index.html

- CDC Data Directory https://www.cdc.gov/datainfo.html

- CDC Youth Risk Behavior Surveillance https://www.cdc.gov/yrbs/index.html

- USAID Agency wide portfolio management system https://dis.usaid.gov/

- Gender Links https://www.genderlinks.org/

- Edu Links https://www.edu-links.org/learning