Compatibility mode isn't what he's talking about. In the developer tools, there's also "browser mode", which allows you to select a different user agent, and "document mode", which allows you to select a rendering engine from IE7 onwards.
Turned out to be an IE7 specific regex bug (a look ahead bug), that we couldn't replicate using IE8 and IE9 set to IE7 compat mode.
Tracked down a copy of XP with IE7 and replicated the bug. Which led me to Google regex bugs related to IE7 and managed to track down the exact issue.