As someone who tries to be prompt to a fault, I can see that yes there are people who get annoyed at promptness. It's not that you're a bad person for being prompt. Rather you're a bad person if you start without them or otherwise push back on their lateness.
But there are people where shit seems to happen more than for others. Late once in a blue moon? No worries. Repeat offender? That's a you problem.
And the late people need to understand that sometimes they will miss the beginnings of things, but that's ok too; their inability to be on time (for whatever reason) should not waste the time of those who get there on time.
So then you waste even more time when someone recaps for them.
It's almost like people need to think about their day when they're scheduling things instead of just accepting every single meeting.
You can request different times for things. That's an option.