Well, to throw in a wildly out of context Einstein quote: "We cannot solve problems by using the same kind of thinking we used when we created them.".

What I'm trying to say: you cannot generate a set of test problems using the program you are trying to test, unless you know the program is correct, which is probably why you want to test it to begin with (unless you are just writing tests to prevent regression). On a side note: if you ever find yourself working on machine learning, you'll find that the same goes for rigorously separating your training set and benchmark/test set).

Formal verification of software is hard. It's a huge field of research. If you really want to prove your algorithm is correct (not your implementation of it!), you're probably better off going the mathematical or formal modeling route. If you want to test your implementation, maybe you don't need to test every single case. A set of carefully hand-crafted test cases might go a long way.