A few things:

x86 is harder to decode than arm since the instructions are anywhere between 1-15 bytes while arm instructions are 4 bytes each. It has quite an effect if you want to decode more than 1 instruction per cycle.

x86 has a stronger memory model than arm, which will come at an extra cost if implementing any sort of out-of-order execution. I'm not very familiar with how it affects the cache implementation.

The x86 instruction set is more complicated than arm, and instructions frequently decode into multiple uops. I believe that microcode is used more extensively in x86.