Blacklist and whitelist are also not racial either but have been "cancelled" by some.

"the black ball signifies opposition"

Blacklist signifies exclusion.

What makes one racist and the other not?

Anything can be deemed "racial" today. "blacklist" wasn't racial either until it was deemed racial by corporations like GitHub.