In both cases we can only fix a particular instance of an error and effectively have no way to show that similar errors don't exist, or even that we didn't introduce new bugs with the fix.
My point was not something about manipulating weights directly vs. augmenting the training set. The analogy is roughly, find a misclassification : add particular input to training set :: find a bug : add a patch/test case. I'm playing devil's advocate, of course, and there are obvious, important differences.
> The difference is that all human written systems was once understood by a human
But that's not one of them! In practice, we create software systems by getting some rough intuition about an interface, wiring together some components, then checking the results empirically. Basic components that everything else is built on, C compilers for instance, are based on ambiguous, even contradictory specifications.
Unfortunately, adding that particular input to the training set may mess up the ANN when given a different input (i.e. you fix one bug but create one or more new bugs, like a game of whackamole but where people die)
Can we actually do this, for complex ANNs?
Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).