Calhoun college was so named in 1931, and the only reason seems to be that he was a Yale alumnus who was Vice President. I'm not seeing the big offsetting factors. It's not like they're trying to rename something named after Thomas Jefferson, who also owned slaves.