x86 is harder to decode than arm since the instructions are anywhere between 1-15 bytes while arm instructions are 4 bytes each. It has quite an effect if you want to decode more than 1 instruction per cycle.
x86 has a stronger memory model than arm, which will come at an extra cost if implementing any sort of out-of-order execution. I'm not very familiar with how it affects the cache implementation.
The x86 instruction set is more complicated than arm, and instructions frequently decode into multiple uops. I believe that microcode is used more extensively in x86.
ARM also offer a selection of lower-end devices which have both lower MIPS and lower watts, especially in standby. Intel don't have anything to match. You can't buy a $1 Intel device that draws microamps in standby and comes with a selection of useful microcontroller peripherals.