For practical purposes it should suffice if they create an exhaustive list of the encryption software that is used, say, in 99% of all cases. Finding those and documenting some vulnerabilities of each is quite a realistic goal.

It's very easy to detect those kind of fuck ups, you just have to look for them. We don't have the mean to do it, and sometime we are just ignorant or lazy. They are neither.

We assume actual human beings need to press buttons to detect obvious developer errors. I bet if an encrypted communication with any kind of bad (or even just popular default) parameter goes through anything the NSA oversees it gets instantly attacked and put in a bin somewhere with your new software name on it. The machine probably even picks a random name for the exploit once it found one.

The NSA is old enough, and well resourced enough. Time and money solve these problems readily enough. In the case of FOSS, all you really need is a parser designed to hunt down specific sorts of flaws. We have static analysis for non-crypto needs, it seems reasonable someone at NSA got funded to write one for their use case.

If they have a complete IP traffic recording facility, they will be applying automatic classification to it and looking to make the classification as complete as possible. Anything that doesn't fit the existing categories will attract attention.