Hacker News Reader

Comment by varjag | Hacker News Reader

That's the thing about neural networks: any QA is going to be superficial due to their statistical black box nature.

Exactly. It's the same reason that no amount of unit tests can replace formal methods for safety-critical software, and we cannot apply formal methods to neural nets [yet].

That's not really true. Most safety critical software is tested without formal verification. They are just really really thorough and rigorous.

Formal verification is obviously better if you can do it. But it's still really really difficult, and plenty of software simply can't be formally verified. Even in hardware where the problem is a lot easier we've only recently got the technology to formally verify a lot of things, and plenty of things are still out of reach.

And even if you do formally verify some software it doesn't guarantee it is free of bugs.

What? Black box testing has plenty of techniques: https://en.wikipedia.org/wiki/Black-box_testing

Whether it's a neural network inside or not is completely irrelevant. That's why it's called "black box".

This seems to ignore that if you look inside the box at code you could understand it whereas looking at the activation values is unlikely to illuminate.

Practical neural networks operate in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs. Your FSD could well recognize stop signs in your battery of tests but not when someone drew a squirrel on it with green sharpie.

Your run of the mill computer program also "operates in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs".

This is hardly similar as the state of a typical computer program can be meaningfully inspected, allowing both useful insights for adversarial test setups and designing comprehensive formal tests.

Right, if you consider the internal state, it is hardly similar. You talked about black box and QA though. Black box by definition holds the internal state as irrelevant, and QA mostly treats the software it tests as a black box, or in other words the tests are "superficial" as you call it.

Black box testing in typical software is, however, less superficial, because the tester can make inferences and predictions about what inputs will affect the results. When you're testing a drawing program, for example, you may not know how the rectangle tool actually works internally, but you can make some pretty educated guesses about what types of inputs and program state affect its behavior. If the whole thing is controlled by a neural network with a staggering amount of internal state, the connections you can draw are much, much more tenuous, and consequently the confidence in your test methodology is significantly harder to come by.

Something a bit similar is clinical trial and it is accepted without problem.

You make a black box test on several thousands (sometimes only hundreds) patients, and if patients who received the drug perform better the patients who received the placebo, then the drug is usually accepted for commercialization.

Yet one isolated patient may be subject to several comorbidities, her environment could be weird, she could ingest other drugs (or coffee, OTC vitamins or even pomelo) without having declared it. In a recent past women were not part of clinical trials because being pregnant makes them very "non-standard'.

First of all, clinical trials are typically longer and more thorough than you imagine, they span years. The fact that COVID vaccines were fast-tracked gives people wrong idea about it.

Secondly, even after the product hits the market the company is still responsible for tracking any possible adverse effects. They have a hotline where a patient or doctor can report it, and every single employee or contractor (including receptionists, cleaning staff, etc.) is taught to report such events through proper internal channels if they accidentaly learn about them.

> clinical trials are typically longer and more thorough than you imagine, they span years

I don't know where you get that, most clinical trials last 26 weeks, even in phase III.

and about "more thorough than you imagine" no, most are subcontracted to CROs and the way clinical trials are conducted is messy.

Below is story from the POV of a PI.

But similarly many patients complain about the way they are treated in visits and the lack of interest of the nurse/doctor who receive them.

https://milkyeggs.com/biology/why-are-clinical-trials-so-exp...

> Something a bit similar is clinical trial and it is accepted without problem.

Clinical trials also have strict ethical oversight and are opt-in. If clinical trials were like Teslas, we'd yeet drugs into mailboxes and see what happened.

Reply on news.ycombinator.com