I've come up with counterexamples for every such formula or rule that people have sent me (usually running a Python loop with gmpy2.is_prime() or something), but unfortunately the existence of the counterexamples somehow didn't occur to them before they wrote in in the first place.

https://en.wikipedia.org/wiki/Mathematical_coincidence is a big problem because people will sometimes conjecture a pattern and then find that it holds for 6 or 7 integers and conclude that it's right. Or if they find a counterexample, they might just make a minimal modification to the conjecture and conclude that they're fixed it! So there are lots of these that will say "unless the number is..." and presumably give some other random property that the counterexample that the claimant found had.

It feels like some people think that mathematics works by people guessing rules and then checking them, and regarding the rules as right if they "check out" in practice. (I know that there's been lots of progress made from guesses and conjectures and numerical experiments; the issue is just what comes after the guess!)