I would agree with you. I never want my manager there. As an EM, I never want to be there. And yet, as the years go by, when I have an EM role I find my presence is requested. Again, who knew?
If the EM is not there, those are not actionable. What we try to do is: do the retro by ourselves, and invite the EM at the end of there is stuff for them. Usually they are not available though...