The gay baker was being asked to engage in an action that's deeply disparaging of his beliefs and values.
This is the problem with social laws. It's easy to see things from our own perspectives, without considering that what 'we' consider perfectly reasonable can be perfectly offensive to another individual. One of the most crucial issues in forcing people to engage in behavior that they find offensive is that it goes both ways.