How would one reach a threshold of precision if the data on which precision is determined is missing or faulty?
This doesn't overcome a true "black swan", but that's not what NN are meant to be doing anyhow.
How do you do this when you cannot verify that your data, in subset or in whole, is accurate? And furthermore you don't know how inaccurate it is?
You train on a subset of the initial data. Even if the data has a certain number of incorrect frames, it should still do a decent job getting a lot of things right.
Then you manually loop through all the images of the data set for which the network has detected something that isn't present in the annotations (and vice versa). If the network correctly identified a missing item that wasn't in the original set, all you need to do is press "correct" (and, again, vice versa). You now have an improved data set.
Retrain, rinse, repeat.
Eventually, you'll converge to a case where you have consistency between training and annotations. And then, you manually go through all images again to weed out the final mistakes.
The benefit of this method is that it's much faster to click "correct" that it is to draw rectangles on the screen to label something.
You would be far better off paying people $x/hour to look at an image for 30 seconds and answer the question "Does this image have any people in it?" Y/N and watching for the human who says "Yes" when the AI says "No".
Their accuracy rating will help distinguish who is best able to detect pedestrians that AI and other people missed (and who is just random-clicking Y/N for pay), and their group effort will ensure that someone eventually sees the pedestrian, even if no one else has.
Asking them to draw boxes distracts them from their job, which is "verify that we are able to detect human beings with perfect accuracy vs. a hundred people trying to detect human beings".
(At worst, ask them to click on the person. No need for a box. Either it's a person or it isn't. If it is, and your AI missed it, then what they think is the right kind of box to draw is the least of your concerns.)
Google must have a really good data set from all those "click the boxes containing X" tests they make people do.
^ "the pricing of goods or services at such a low level that other suppliers cannot compete and are forced to leave the market", to quote Google's definition (that was itself taken from some other company's dataset).
As I understand it, this is the core of recursive Bayesian estimation. At the end of the day we don't really have ground truth for anything - it's all filtered through senses with error bars. So any learning process needs to be robust to that.
If you have no idea of the accuracy of any of your data then you're probably asking the wrong question of it. You can do things like test for consistency using cross-validation, e.g. does half of your dataset predict the other half with the same kind of performance? But that can't detect the same errors repeated throughout your data.