Even there, though, one can make educated guesses, based for example on how many cases come up where people got away with it at the time, but then evidence comes to our attention later and we can figure it out in retrospect. Such cases are useful because then we can extend our software to catch them in the future.
I'd never claim that such manipulations never work; obviously we can't know that. It's possible that superclever manipulators are rolling HN in their hands like a piece of silly putty. All I'm saying is that if "most of the stuff on top of HN comes from [that]", then I don't know the first thing about my job.