IP law lets software have the best-of-both-worlds. Source code is patentable & protected by copyright, but is also a trade-secret (because it's not publicly disclosed).
This is problematic when these black-box algorithms are used to assist in the processing of legal cases.
-----
My 2 cents:
- The overall concern that our lives are being governed by the invisible hand of black-box abstractions really resonates with me. Richard Stallman has been warning us about this for a while [0], and I've personally started considering my role in this as a software engineer [1], although I don't have any actionable advice yet.
- I don't feel as strongly that all of this code should be open-source. I think it might be sufficient to just strictly define a series of tests -- measure the inputs & outputs, describe the requirements... that kind of thing.
It's inherently difficult whenever there's a constantly-evolving ML model behind the curtain. It reminds me of the issues with using AI in the aviation industry, for instance [2]. As I recall from my undergraduate classes, aviation control systems have been unable to use "fuzzy" logic to pilot planes due to strict deterministic testing requirements.
[0] https://www.gnu.org/philosophy/free-software-even-more-impor...
[1] https://gazzini.com/essays/posts/tools/
[2] https://www.nlr.org/areas-of-change/introduction-non-determi...