There is a sense in which that is a great idea, and a sense in which it is a not so great idea. If AlphaProof could prove theorems as superhumanly as AlphaGo wins at Go, that would really be something, they might be able to claim all of the millennium prizes. However, the longterm usefulness to mathematics may be somewhat limited. Mathematicians are interested in proofs not so much because of the statements, but because of what those proofs add to our knowledge of the way math fits together. It isn't just about "true" or "false," it's about why. If AlphaProof generated a long and unreadable proof with no discernible structure, it would not be such a large contribution to mathematics as if a human figured out "why" and wrote it in a way other humans could understand.
Of course, it is possible that clever users could tease useful information out of the black box truth oracle. Perhaps by modifying the statement of the conjecture and checking each modification, they could discover some parts of the structure of the problem, which would help inspire a useful proof. Furthermore, it would be helpful to never again waste time trying to prove a false conjecture.