Common mistakes: Testing with xdebug enabled in one version or testing with opcache enabled in only one version.
PHP 5.6.0beta4 (cli) (built: Jun 12 2014 14:13:53)
Zend Engine v2.6.0-dev, Copyright (c) 1998-2014 Zend Technologies
with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
with Xdebug v2.2.5, Copyright (c) 2002-2014, by Derick Rethans
vs.
PHP 5.5.13 (cli) (built: Jun 12 2014 13:54:54)
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
with Xdebug v2.2.5, Copyright (c) 2002-2014, by Derick Rethans
...but, as I said - this is with very limited testing, basically just on Travis (which goes up and down in speed all day, but the difference between versions is pretty steady).
Is xdebug load neutral? Maybe you need to skip loading that during testing, you wouldn't use it during production?