The obvious solution would be, just don't send data to the player's client about enemies that are behind walls. But this is a surprisingly hard thing to engineer in realtime games without breaking the player experience (see: https://technology.riotgames.com/news/demolishing-wallhacks-..., and then notice that even in the final video wallhacks are still possible, they're just more delayed).
With respect I'd like to disagree on this subtly. A lot of games have the client send their cursor position at relatively frequent updates/packages (i.e. sub-second). So the server knows pretty precisely in which direction and to which object a player is looking.
This in turn can be readily used upon when using wall-hacks, as most players, who use wall-hacks tend to almost faithfully follow objects behind walls with their cursor, which good moderators can usually spot within a few seconds, when reviewing such footage (source: I was involved in recognizing Wall-Hacks in Minecraft, where players would replace textures, to easily find and mine diamonds underground).
In blackjack card counting is (probabilistically) caught by tracking player winnings. If someone is beating the odds a bit too much it's a fairly good indicator they are counting cards. Of course in this case getting it wrong isn't so bad from the casinos perspective either since then they'll just kick out a player that was costing them money anyhow.
When the enigma cypher got cracked they had to be very careful about when to act on information gained. If they started beating the odds too much the Germans would cotton on to enigma being broken.
My point being that cheating will almost by definition improve your odds. There are definitely ways to catch that sort of thing happening without installing rootkits. You just might need to hire a couple mathematicians to figure it out.
Examples would be pre-aiming corners and >99th percentile reaction time.
You need a false positive rate well below 1%.
It's basically the usual cat-and-mouse game of an arms race.