It sure looks like such models are going to have to undergo the same sort of scrutiny regular software does nowadays. No more closed-off and rationed access to the near-bleeding-edge.
With companies like Atlassian just going down and not coming, one wonders whether the concept of a technical Ponzi Scheme and technical collapse might be the next thing after technical and it seems like the fragile ML would more accelerate than stop such a scenario.
Edit: Now that I think about it, can't data poisoning happen when predicting, rather than just happening in the training phase? In that case, it's going to be complicated to work around that.