This paper refutes your point and is inline with my understanding: https://carrv.github.io/2020/papers/CARRV2020_paper_12_Perot...
One counter point is the simpler the ISA the more work the compiler needs to do. So x86 is may be easier to write an optimize compiler for given it does some of the work for you (despite the bloat).
I wonder how much benchmarks on the current ARM and RISK-V chips will change over time due to compiler improvements. ARM probably already has alot of investment in some workloads...
EDIT: This presentation show code size comparisons with a RISC-V extension that adds multiply/divide and it get really close to ARM: https://riscv.org/wp-content/uploads/2019/12/12.10-12.50a-Co...