Inside the loop, multiply the matrix with vectors for which you know the answer, and keep making sure the answer is correct. Only when the tests pass should this matrix' product be used for any other API.
6 months later: The AI successfully grey goos the Earth into trillions of paperclips.
We'll do this because the jobs we want the program to do will be too complex for us to write, so we'll write programs that can write the programs. We won't understand the result directly.
This is of course the usually accepted route to the singularity.
If it looks like it produces the right answers, it's done. If you ask why it fails on this case, you get a shrug. If you want to know if it will keep working, you get a blank look, like you are speaking Etruscan. If you ask, "So, how does it work?" you get a paragraph on the basics of matrix multiplication.
Which can be summed up as "it compiles, ship it!"
Suppose you are working on a superintelligent AI, and your tests catch it destroying the contents of its sandbox. You fix that bug and move on to the next failing test, fix that too, and so on. When you run out of tests, you write more and fix those that fail, and so on. Eventually you can't come up with any failing tests anymore.
When that happens, is the AI safe? I don't know. It passes all the tests, but all the previous versions had some bugs, so who is to say that this version is correct? Maybe your test scenarios just aren't realistic enough. Maybe they are realistic enough, but you haven't been looking hard enough for undesirable behavior.
To run a superintelligent AI with any significant access to the outside world, I would like to see much stronger correctness guarantees than just a test suite, because tests can only show the presence of bugs, never prove their absence (paraphrasing Dijkstra).
Reply to Smaug123: The brain is mostly deterministic. Quantum-level nondeterminism has very low probability to matter much.
The brain is likely to be very debuggable. Even today, we continue making progress figuring out more and more things about it. If we had designed the brain, we would know its functioning almost entirely.
You know that we can understand smaller animal brain very well now, right? There are projects going on right now which aim to perfectly simulate a worm's brain. We didn't even design a worm's brain, and we would design the AI and know all its evolution principles, and have infinite access to all its internals (which we don't have with the human brain).
Reply no. 2:
Since you guys down-voted me for disagreeing with your prophet, HN rate-limited my ability to comment.
"Knowing the functioning of something" is nothing like the same as understanding why it does what it does. I work on a large operating system, which was designed by humans; it takes many days to find the root cause of bugs. I have the best access it's possible to get to that operating system, and it still takes a large amount of time to even detect that there has been a bug, let alone determine why it happened and fix it. That amount of time to detect a bug could be lethal in the case that the bug arose in a superintelligent being.
That deserves a downvote. Please consider that people might have downvoted you because they believe that what you're saying is wrong; not for any meta-reason.
Sources, please?
At an atomic level, events are not deterministic.
At a molecular level, events are not deterministic.
Perhaps you are referring to parts of the brain that are not made of atoms?
Personally, I suspect that the brain is entirely non-deterministic in terms of the material it is made of and fine-grained behaviour [because physics and chemistry support that idea].
But I suspect the brain's structure does something to offset this at the larger scale, and make it more predictable/deterministic (averaging, wisdom of crowds, etc). However, I do not assert the latter as fact, merely my own speculation.
At a molecular level, events are not deterministic.
Therefore, we all behave randomly. Qed.
This doesn't follow. The combination of many random events can be highly predictable.
Tossing a fair coin is random. Combining 10000 coin tosses will give you a result close to 5000 heads and 5000 tails, and I can predict you won't get 4000 heads and 6000 tails or 100 heads and 9900 tails.
So there's various types of randomness, and the more you average out uncorrelated events the more you get something very close to a predictable result.
Example via A. Einstein:
"The probabilities are what they are, no matter what we want, no matter what we visualize, no matter what we meditate upon while imbibing inspirational substances. It doesn’t matter how much you want that electron to be spin up, it doesn’t matter how many good “vibes” you give the electron about being spin up, it always has exactly a 1/2 chance of being spin up, a 1/2 chance of being spin down."
Quantum events in the brain would decohere almost immediately, leading to effectively nondeterministic behavior at any scale above the cellular. The philosophical determinism of wavefunction realism (which is the LW argument) is irrelevant to the actual behavior.