They derail the conversation. Of course they're annoying..
A great example is "red-head scarf"; what you read is "(read head) scarf" what it means is "red (head scarf)". Granted, it makes the wording unnecessarily complicated, but it's probably grammatically correct.