I don't think it's fair to compel people to do things they don't want to do. If someone does run their business like your example, let society decide whether to continue supporting them. The government should not be able to force people to work.
The government is not telling you to sell cakes on your day off; it's saying that as long as you're already selling cakes for the public, you can't be a bigot and discriminate.
Consider a case in which a majority of society excludes a minority from most commerce. Are ghettos, ‘separate but equal,’ and whites-only fountains not worth preventing?
That reads to me as "I don't think laws are fair and people should be able to do whatever they want to do."
I don't want to pay taxes, I don't want to pay my bills, and I don't want to have to wait to be sober to drive, but I do it all anyways because the law compels me.
I appreciate that physically hurting is not appropriate in your law, but it should be your right to hurt others in any non-physical manner if you feel like it, no limits.
Dropping concrete limitations to the scope of law opens it up too anything. What is non-physical harm? What if turning down someone's sexual advances hurts their self-esteem? What if refusing to work for me hurts my ability to make a living.
The point is that once you abandon a concrete definition of harm, your punished into a world of capricious law what's the times to regulate every individual Behavior on a Case by case basis.
But that is what underpins the concept of business.
A set of restrictions on what activities a business can do and how they interact with society.
To get shocked when society wants to prevent discrimination seems odd.