Code-gov: A collection point for all Code.gov repositories | Hacker News Reader

Code-gov: A collection point for all Code.gov repositories

github.com

93 pointsjohncole16 comments

Dead repo? Looks like it hasn't been updated in over 3 years. There's a few PRs that are pretty old that haven't been merged either.

They still have a ton of active repos[0], but yeah, this one seems dead; all of the repos it links to in the readme are archived or deleted.

[0]https://github.com/GSA

Let's put something on GitHub to distract some computer nerds.

Shared this why?

Seems a more accurate source, tho harder to search through: https://code.gov/agencies

That page also hasn’t been updated in years.

GSA used to maintain a combined catalog that was refreshed a few times per year and searchable.

I’m also not sure if there’s still a requirement for agencies to keep their own code.json up to date. It’s hard to tell when each department refreshes, but it seems like HHS hasn’t updated theirs since March of 2022.

Do people like incentives or penalties?

What incentive could there be to keep reusable [federal] open source software inventoried?

I think incentives work better.

In this case, GSA used to scrape and combine and then stopped. Also GSA used to ask agencies to update their inventory and then stopped.

I think if GSA just asked, it would increase the recency and completeness of the code.jsons.

Also, what’s kind of funny is that since open source projects, by their nature, are publicly visible, GSA could probably just scrape and combine together and not rely on lots of different agencies to have their own processes.

"Git scraping: track changes over time by scraping to a Git repository" (2020) https://simonwillison.net/2020/Oct/9/git-scraping/ :

> Every 20 minutes it grabs the latest copy of that JSON endpoint, pretty-prints it (for diff readability) using jq and commits it back to the repo if it has changed.

> This means I now have a commit log of changes to that information

A static site builder can rebuild just the pages of the site that need to be changed once in a Github Action that updates the site when a Pull Request is merged to main.

Though, if the data quality is insufficient because the data sources are not updated, then downstream apps and static sites that depend upon the data are also insufficient.

There are ways to do this, but GSA just doesn’t do it.

Years ago they used to have a system that would combine all the code.jsons into a single db and provide a query interface. They stopped funding that system and redesigned this static site. But could have used GitHub actions or something to fetch and combine the code.jsons and do everything client side. That still wouldn’t have needed maintenance costs.

Here's a way to scrape URLs to JSON/YAML and then build static HTML with Hugo in a GitHub Action: https://github.com/jackyzha0/hugo-obsidian

datasette is a webapp and CLI built on SQLite and Python. datasette-lite is the pyodide + WebAssembly build of datasette which can be served as static HTML, JS, and WASM SQlite.

datasette: https://github.com/simonw/datasette :

> Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.

> Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world.

From "Deploying a live Datasette demo when the tests pass" (2022) https://til.simonwillison.net/github-actions/deploy-live-dem... :

  datasette publish vercel fixtures.db [...]

The `datasette publish` command supports Google Cloud Run, Heroku, Vercel, Fly, [Full Container or Serverless] https://docs.datasette.io/en/stable/publish.html

datasette-lite: https://github.com/simonw/datasette-lite :

> You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*

> [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you

> To load a Parquet file, pass a URL to `?parquet=`

> [...] https://lite.datasette.io/?parquet=https://github.com/Terada...*

There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.

datasette plugins are written in Python and/or JS w/ pluggy: https://docs.datasette.io/en/stable/plugins.html https://datasette.io/plugins

datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server.

datasette-scraper/#architecture: https://github.com/cldellow/datasette-scraper/#architecture

(TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark: https://github.com/rushter/selectolax#simple-benchmark )

(Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct )

datasette-graphql adds a GraphQL HTTP API to a SQLite database: https://datasette.io/plugins/datasette-graphql

plugins?q=sqlite: https://datasette.io/plugins?q=sqlite

datasette-sqlite-fts4: https://datasette.io/plugins/datasette-sqlite-fts4 ; Full-Text Search with SQLite

datasette-ripgrep: "deploy a regular expression search engine for your source code": https://github.com/simonw/datasette-ripgrep

Seeing as there's already a JSONLD @context (schema) for code.json, CSVW as JSONLD and/or YAMLLD would be an easy way merge Linked Data graphs of tabular data: https://github.com/semantalytics/awesome-semantic-web#csvw

A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.

There are code.json generators and a scraper written in Python:

GSA/code-gov//docs/code_json_generators.md > "Code.gov Metadata Schema 2.0.0 Requirements" https://github.com/GSA/code-gov/blob/master/docs/code_json_g...

code.gov/agency-compliance/compliance/procurement: https://code.gov/agency-compliance/compliance/procurement

Looks like there's a JSON-LD @context for /data.json now: https://project-open-data.cio.gov/v1.1/metadata-resources/

Prometheus-like data pull system might've been better for COVID reporting w/ e.g. the hastily-added CDCPMDRecord and SpecialAnnouncement.

Prometheus: https://en.wikipedia.org/wiki/Prometheus_(software)

CDCPMDRecord: https://schema.org/CDCPMDRecord

"git-scraping" https://www.google.com/search?q=git+scraping

gh topic: git-scraping: https://github.com/topics/git-scraping

At EPA we use this to keep this up to date but it just scrapes our GitHub:

https://github.com/USEPA/code-json-generator

This code.gov initative comes from Obama-era push to use/release open source, but the attention now seems to be on data (data.gov) and ai (ai.gov)

this is a preview of "pass laws on gov and civilian software reporting, this is how we (gov) do it" ?

chilling, dysfunctional.. almost mocks the daily and weekly maintenance that is rigorously required in so many competent organizations.

combined with new pending legislation, quite the stark warning IMHO

What pending legislation?

I wish I could list it all .. among people I talk to, it is the EU CRA that is most pressing.. there are many more

https://www.eff.org/deeplinks/2023/05/eus-proposed-cyber-res...

Reply on news.ycombinator.com