That's precisely what's at issue. Who decides who should be celebrated and who should not? What are the criteria for deciding? And what story are we telling about our history with those decisions.
in this case, the leadership at Caius College. in every other case, the same sort of people who made the decision to celebrate the individual in the first place.
in a broader sense, history does - the same people who rail against taking down statues seem to like talking about how history will view the leaders of today. "cancelling" historical figures is the literal implementation of "history will judge my actions". history has judged this guy, and decided that eugenics aren't cool.