The wear patterns of jeans aren’t good forensic evidence

What they found was a correlation between false positive rates and true positive rates, where IF you could accept a false positive rate of 0.1%, THEN you could get a little over 50% true positive rate with the longest seam length under "ideal imaging conditions".

They do not attempt to estimate the false positive rate of whatever methodology FBI agents are using in practice. I doubt the FBI has a rigorous well-defined test that could be analyzed; I suspect this is just a general technique that is applied by different agents in different ways.

Even if the FBI's methods approach this ideal false positive rate (which they can't), they are not describing it honestly in court.

> When, for example, an FBI analyst identified the plaid shirt in a surveillance video as that belonging to Wilbert McKreith, the analyst stated in court that the odds that two different shirts would match were a staggering 1-in-650 billion. The analyst arrived at this number by making eight measurements along two seams, estimating the probability that the plaid stripes at these eight locations were misaligned by a certain distance, and then multiplying all eight of these probabilities to reach the astronom-ically low odds of 1-in-650 billion.

>The analyst excluded any seams that did not satisfy each of the following properties: 1) the length of the seam was at least 30 cm, 2) the resolution of this 30-cm seam was at least 1,500 pixels, 3) the width of the seam was at least 30 pixels, and 4) the final extracted 1D signal clearly showed a distinct pattern of ridges and valleys.

Even with photos taken specifically to see these patterns, "[e]nforcement of these criteria reduced the original 211 left/right and inner/outer seams to 164 left/inner, 162 right/inner, 81 left/outer, and 90 right/outer seams."

Unfortunately they aren't very clear on what they mean by different orientations/lighting conditions/drapes.

1 person committed a crime. So the true positive rate is 1 out of 7 billion or so. So how many false positives would 0.1% lead to? About 7 million pants wearers?

And for false negatives, we expect half the time the true criminal's pants would prove them innocent?

I don't think you can really trust evidence if the true criminal has a coin-flip chance to show innocence with it.

https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/

Without confirmation, false positive rates varied from 2 to 4 percent. That means up to 4 percent of test subjects will end up arrested because of a false positive.

If a piece of evidence is used to identify, you select for the population that matches that property.

A common certification at the time was earned by taking an online class and answering multiple choice questions ... and blamo you had a certificate that they found on a lot of resumes. It didn't require much of any actual scientific knowledge.

If one judge somewhere allows a prosecutor to present junk science as evidence, that becomes precedent that other prosecutors can point to in order to convince other judges to allow it, too.

Once precedent is established, it takes the decision of a higher court to stop use of that junk science. It takes time for cases to work their way up.

We need tooling to better understand when people have been wrongfully convicted using inaccurate forensics sciences so they can more quickly get their case re-evaluated (and speed exoneration if warranted). If someone from Code For America sees this, consider such a tool (similar to what has been built to rapidly expunge marijuana convictions in states that have legalized it).

Interesting and on the side of caution is that false negatives are much much higher than false positives in this Jeans pattern matching.

From just reading the Wiki article (key caveat), I would say the case is not "beyond reasonable doubt" thus a conviction is unwarranted. But that doesn't mean everything points to a tragic incident. This is a pretty murky case.