Hacker News Reader

Comment by barking_biscuit | Hacker News Reader

>If you want to trust a prediction, you need to understand how all the computations work.

I disagree with the premise here. You don't understand how all the computations work in the brains of people's predictions whom you trust. You simply have a mental calculation of their batting average through exposure to their track record and this batting average functions as a proxy for trust.

I find this is more or less the same way that I learn whether or not I can rely on GPT-4 for a particular use-case. If it's batting average is north of a certain % for a given use-case, then it doesn't need to be right 100% of the time for me to derive value from relying on it.

I think we are slowly crossing a threshold where we accept indeterminism and mistakes from machines in a way that we haven't in the past.

View on HN

I disagree too (despite being a researcher in the general neighborhood), but I find the usual comparison with a human brain to be inaccurate. We don’t trust a person just because of their track record, the fact that they are human to begin with already comes with a lot of baggage. I think the best analogy is with a very complicated piece of hardware.

That said, someone has to have a good understanding of what is happening (not necessarily the end user). In my experience neural networks today are basically like compiled software which source code has been lost. Every act of debugging and fixing has to include also reverse engineering (I’m talking about the actual network weights, not the e.g. PyTorch code). It’s too inefficient and cumbersome, and as they become more and more common, statistics says the failures will become more often and more severe.

When do we, for example, decide to put the first NN on an airplane cockpit?

The point is that you trust those people. You can't trust a black-box model, but you can trust a result if the explanation that is provided actually explains it. That is, if the explanation has factors X, Y, and Z, then every time those hold the result should be the same.

Otherwise, you are just trusting the training set and the training process. Sometimes that's fine, as the article also mentions.

No, the point is those people are also a black-box to you and they are a black-box that you trust because of YOUR training set and training process.

Yes, your exact thought process may be a black box to me, but I myself am a black box made to the same (DNA etc) spec. We have millions of ancestral years of experience trusting each other and we know how to simulate each other because we are the same kind of black box. Your trust in other humans is explained much more by your priors over human behavior than by any observed behavior of individual humans--that's layered on top.

In short: we trust people because we can model them effectively, and being one helps. We trust mathematical models and traditional programs because we understand them and can check their work in any specific instance. Large ANNs don't (yet) benefit from either of these two kinds of trust.

They may be referring more to a theory of mind than a theory of neuro-physics. I trust the decisions and opinions of those whose reasoning capacities I trust. It's not enough to know what they decide, you must also know the set of choices in front of them at the time and their process for choosing one among them.

Reply on news.ycombinator.com