One reason employees don't want these conversations is that they have a shitty manager.

But it can also be that they're a shitty employee. The chance of "saying the wrong thing" is much bigger if you are in fact the wrong person for the job and/or team.