They caused problems by not leaving when they were asked it. I've seen white people be asked to leave and they've left without causing problems.
Not all of us like to respect unjustified requests, especially when there's a reason for doing what we're doing.
You leave when requested or you are trespassing. It is that simple.
Even buying a coffee doesn't entitle you to stay against the owner's wish.