The right analogy is cryptography. You don't need to understand the math behind cryptography to calculate a hash, or encrypt something. You just need to understand at a very high level what is asymmetric encryption, IV, etc.
That’s actually a bad analogy in my opinion. People who don’t learn cryptography make many mistakes calling cryptography libraries and those mistakes can’t be uncovered by testing.
Oh, it'll all be great when ChatGPT writes all that code for us and we can be off somewhere having a beer and a cry instead.
You'll cry but, to listen to your sorrow and offer friendly support, ChatGPT will be there for you !
That makes me think of a chapter in a Daniel Suarez novel where an AI tortures a human for a prolonged period of time.
Learning how to use cryptography responsibly is different from learning to design and attack cryptographic algorithms, and fortunately a lot easier.
People can make ML mistakes if they don't learn ML. Some of the mistakes can be uncovered by testing, but only if you know enough about ML to know how to test it. Some mistakes just you with a bad model that could have been better.
I think what we all want is some kind of blackbox where data goes in and insight comes out. Ideally we would have something that takes a bunch of data and finds the shortest program that generates it. Unfortunately such a black box cannot exist, it is uncomputable. So then we need to ask for something less than that and we don't know exactly what that is.
Maybe not the shortest program, but perhaps a list that contains the shortest program:
Interesting but how would one use such a list to make predictions?
If the data was generated from a generator of constant complexity, its kolmogorov complexity cannot exceed the complexity of the generator. As the amount of data increases, the kolmogorov complexity will increase until it converges after a finite amount of data. At this point we effectively have the generator so we can generate infinite data.