That is one kind of problem. The 'Genie' problem: be very specific what you wish for because the genie will misinterpret you.
But it's not the only kind of problem. There are inherent biases in evolutionary systems. One example: in systems with varying length genotypes there is a massive pressure to bloat. Even if you encode 'small' as a strong requirement in your fitness function, it may not be enough (or it may be enough to completely defeat whatever your real goal was).
There are inherent biases. In the (simulated) genetics, in the genotype to phenotype mapping, in the evolutionary operators, even if you get the fitness function right.
Evolutionary computing as an engineering tool is hard.